Skip to content

Ingestion

Installed with pip install 'credencegraph[ingest]'. read_latex(path) reads a .tex document, and the files it inputs, into Chunk objects located by file and line; chunks_to_graph(chunks, document) turns them into a graph of anchored document nodes with cites relations. The ingest command does both; see Ingesting a LaTeX document.

The ingestion layer: readers that turn a document into chunks, and the graph of document nodes built from them.

This package is installed with the ingest extra. The core never imports it. chunks needs no extra dependency; latex, the LaTeX reader, needs pylatexenc.

Chunk dataclass

A chunk of a document, as a reader found it.

Attributes:

Name Type Description
unit str

One of UNITS.
text str

The chunk's text, normalised with normalise_text.
locator str

Where it is, such as file=sec/method.tex;lines=41-42.
cites tuple[str, ...]

The bibliography keys the chunk cites, in order of first citation.
labels tuple[str, ...]

An equation's labels, in order.
key str | None

A bibliography entry's key.
Source code in src/credencegraph/ingest/chunks.py 
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
@dataclass(frozen=True, slots=True)
class Chunk:
    """A chunk of a document, as a reader found it.

    Attributes:
        unit: One of ``UNITS``.
        text: The chunk's text, normalised with ``normalise_text``.
        locator: Where it is, such as ``file=sec/method.tex;lines=41-42``.
        cites: The bibliography keys the chunk cites, in order of first citation.
        labels: An equation's labels, in order.
        key: A bibliography entry's key.
    """

    unit: str
    text: str
    locator: str
    cites: tuple[str, ...] = ()
    labels: tuple[str, ...] = ()
    key: str | None = None

chunks_to_graph(chunks, document, initial=None)

Build the graph of document nodes for a document's chunks.

Every chunk becomes a node anchored verbatim: its statement and its anchor's quote are the chunk's text, and the anchor's digest is the text's digest. An equation with labels carries attributes.label: its label, or the list of them when it has several. A bibliography entry carries attributes.key. Each citation becomes a cites relation from the citing chunk to the first entry with that key.

Parameters:

Name Type Description Default
chunks Iterable[Chunk]

The chunks, in document order.

 required
document str

The document id of every anchor.

 required
initial Mapping[str, str] | None

The type each unit starts with, by unit, as a rubric's initial gives it; any other unit starts unassigned.

 None

Returns:

Type Description
tuple[Graph, list[str]]

The graph, and the cited keys that name no bibliography entry, in order of first citation.
Source code in src/credencegraph/ingest/chunks.py 
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
def chunks_to_graph(
    chunks: Iterable[Chunk], document: str, initial: Mapping[str, str] | None = None
) -> tuple[Graph, list[str]]:
    """Build the graph of document nodes for a document's chunks.

    Every chunk becomes a node anchored verbatim: its statement and its anchor's quote are the
    chunk's text, and the anchor's digest is the text's digest. An equation with labels carries
    ``attributes.label``: its label, or the list of them when it has several. A bibliography entry
    carries ``attributes.key``. Each citation becomes a ``cites`` relation from the citing chunk to
    the first entry with that key.

    Args:
        chunks: The chunks, in document order.
        document: The document id of every anchor.
        initial: The type each unit starts with, by unit, as a rubric's ``initial`` gives it; any
            other unit starts ``unassigned``.

    Returns:
        The graph, and the cited keys that name no bibliography entry, in order of first citation.
    """
    chunks = list(chunks)
    initial = initial or {}
    ids = _ids(chunks)
    graph = Graph()
    references: dict[str, str] = {}
    for node_id, chunk in zip(ids, chunks, strict=True):
        text = normalise_text(chunk.text)
        attributes: dict[str, object] = {"origin": DOCUMENT, "unit": chunk.unit}
        if chunk.labels:
            attributes["label"] = chunk.labels[0] if len(chunk.labels) == 1 else list(chunk.labels)
        if chunk.key is not None:
            attributes["key"] = chunk.key
            references.setdefault(chunk.key, node_id)
        anchor = SourceAnchor(document, chunk.locator, text, text_digest(text))
        graph.add_node(Node(node_id, initial.get(chunk.unit, UNASSIGNED), text, (anchor,), attributes=attributes))
    unresolved: list[str] = []
    for node_id, chunk in zip(ids, chunks, strict=True):
        for key in chunk.cites:
            target = references.get(key)
            if target is None:
                if key not in unresolved:
                    unresolved.append(key)
                continue
            relation_id = f"{node_id}-{CITES}-{target}"
            if relation_id not in graph.relations:
                graph.add_relation(Relation(relation_id, CITES, node_id, target))
    return graph, unresolved

The LaTeX reader: a .tex document, with the files it inputs, as chunks located by file and line.

The reader parses with pylatexenc and keeps the LaTeX source of each chunk, so inline mathematics, \ref and \cite stay in the text as written. It

  • follows \input and \include, relative to the main file's directory;
  • expands the macros defined with \newcommand, \renewcommand, \providecommand and \def (with #1-style parameters), wherever they are defined; a macro still undefined, or nested deeper than MAX_EXPANSION_DEPTH, stays as written;
  • splits running prose into sentences, at ., ? or ! followed by a space and a word not starting in lower case, except after a common abbreviation or an initial, and at paragraph breaks, headings, list items and displays;
  • makes each display (\[ \], $$ $$, equation, align, ...) an equation chunk with its \label values, each \caption a caption chunk, each \footnote a footnote chunk and each \bibitem of a thebibliography a reference chunk; the bibliography is read from the document, or from <main>.bbl beside the main file when the document uses \bibliography;
  • drops comments, \label and headings from the text.

Locators are file=<path relative to the main file's directory>;lines=<first>-<last>.

LatexError

Bases: Exception

A LaTeX document cannot be read: it inputs itself, or a \def body is never closed.

Source code in src/credencegraph/ingest/latex.py 
141
142
class LatexError(Exception):
    r"""A LaTeX document cannot be read: it inputs itself, or a ``\def`` body is never closed."""

read_latex(path)

Read a LaTeX document into chunks, in document order.

Parameters:

Name Type Description Default
path Path

The main .tex file.

 required

Returns:

Type Description
list[Chunk]

The chunks.

Raises:

Type Description
FileNotFoundError

If the main file, or a file it inputs, does not exist.
LatexError

If the document inputs itself.
Source code in src/credencegraph/ingest/latex.py 
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
def read_latex(path: Path) -> list[Chunk]:
    """Read a LaTeX document into chunks, in document order.

    Args:
        path: The main ``.tex`` file.

    Returns:
        The chunks.

    Raises:
        FileNotFoundError: If the main file, or a file it inputs, does not exist.
        LatexError: If the document inputs itself.
    """
    reader = _Reader(Path(path))
    reader.collect_definitions(reader.main)
    reader.install_definitions()
    reader.read_file(reader.main, body_only=True)
    return reader.chunks