{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vnNSrLR5RuqR"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dbamman/anlp25/blob/main/1.words/TokenizePrintedBooks_TODO.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NvMOJqbORuqU"
   },
   "source": [
    "# Tokenization after optical character recognition\n",
    "\n",
    "OCR'd texts present special challenges to tokenization.  Consider this selection from an OCR'd version of Darwin's Origin of Species from the [Internet Archive](https://archive.org/download/originofspecies00darwuoft/originofspecies00darwuoft_djvu.txt):\n",
    "\n",
    "```\n",
    "the inhabitants of the surrounding districts will, also, be thus\n",
    "prevented. Moritz Wagner has lately published an interest-\n",
    "ing essay on this subject, and has shown that the service\n",
    "rendered by isolation in preventing crosses between newly-\n",
    "formed varieties is probably greater even than I supposed.\n",
    "But from reasons already assigned I can by no means agree\n",
    "with this naturalist, that migration and isolation are neces-\n",
    "sary elements for the formation of new species. The im-\n",
    "portance of isolation is likewise great in preventing, after\n",
    "any physical change in the conditions such as of climate ele-\n",
    "vation of the land, &c., the immigration of better adapted or-\n",
    "ganisms; and thus new places in the natural economy of the\n",
    "district will be left open to be filled up by the modification of\n",
    "the old inhabitants. Lastly, isolation will give time for a\n",
    "new variety to be improved at a slow rate ; and this may\n",
    "```\n",
    "\n",
    "Here the printing convention of line-break hyphenization would, under a standard tokenizer, generate incorrect tokens like `interest-ing` (or perhaps `interest-` and `ing`).  **Design a better tokenizer (even just using pre- and post-processing) for these texts.**  Note here the correct tokenization of `interest-ing` is `interesting` but the correct tokenization for `newly-formed` is still `newly-formed`.\n",
    "\n",
    "For a more thorough library for handling OCR'd book data, see https://github.com/tedunderwood/DataMunging\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KQLi1uPHRuqU"
   },
   "outputs": [],
   "source": [
    "# download data\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/main/data/darwin_origin_ia.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6T44DMOxRuqV"
   },
   "outputs": [],
   "source": [
    "import sys, nltk, re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lPniBIjpRuqV"
   },
   "outputs": [],
   "source": [
    "def read_text(filename):\n",
    "    lines=[]\n",
    "    with open(filename, encoding=\"utf-8\") as file:\n",
    "        for line in file:\n",
    "            lines.append(line.rstrip())\n",
    "    return lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "USzVLzyhRuqW"
   },
   "outputs": [],
   "source": [
    "filename=\"darwin_origin_ia.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bu3vKRl-RuqW"
   },
   "outputs": [],
   "source": [
    "full_text = open(filename, encoding=\"utf-8\").read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wOS3qjiuRuqW"
   },
   "outputs": [],
   "source": [
    "test_text = \"\"\"the inhabitants of the surrounding districts will, also, be thus\n",
    "prevented. Moritz Wagner has lately published an interest-\n",
    "ing essay on this subject, and has shown that the service\n",
    "rendered by isolation in preventing crosses between newly-\n",
    "formed varieties is probably greater even than I supposed.\n",
    "But from reasons already assigned I can by no means agree\n",
    "with this naturalist, that migration and isolation are neces-\n",
    "sary elements for the formation of new species. The im-\n",
    "portance of isolation is likewise great in preventing, after\n",
    "any physical change in the conditions such as of climate ele-\n",
    "vation of the land, &c., the immigration of better adapted or-\n",
    "ganisms; and thus new places in the natural economy of the\n",
    "district will be left open to be filled up by the modification of\n",
    "the old inhabitants. Lastly, isolation will give time for a\n",
    "new variety to be improved at a slow rate ; and this may\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Complications with OCR text\n",
    "\n",
    "First take a look through the full text and try to identify some other errors caused by OCR."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the first 500 characters\n",
    "\n",
    "print(full_text[:500])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implement a tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jiPuTb1hRuqW"
   },
   "outputs": [],
   "source": [
    "def ocr_tokenizer(text: str) -> list[str]:\n",
    "    \"\"\"\n",
    "    Takes an input string and returns a list of strings (tokens).\n",
    "    \"\"\"\n",
    "\n",
    "    # Implement me\n",
    "    return str.split(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ocr_tokenizer(test_text)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}