Chunk Documents for Search
The help center's search indexes articles in overlapping chunks of words. The README has the rules for normalizing text and cutting chunks. Build chunks.py with the assistant, and check each window boundary as carefully as the tests do.
chunks.py1 of 3
Files
chunks.py
Python · Editableimport re
def normalize(text):
"""Lowercases the text and removes accents, so "Café" matches "cafe"."""
return text.lower()
def words(text):
"""The words in the normalized text, with punctuation dropped."""
return normalize(text).split()
def chunk(items, size, overlap):
"""Windows of size items. Each starts overlap items before the previous one ends, and the last may be shorter."""
step = size - overlap
raise NotImplementedError
def chunk_document(doc_id, text, size=200, overlap=40):
"""Returns [(chunk_id, chunk_text)] for the search index."""
pieces = chunk(words(text), size, overlap)
return [(f"{doc_id}:{i}", " ".join(piece)) for i, piece in enumerate(pieces)]
import pytest
from chunks import chunk, chunk_document, normalize, words
def joined(chunks):
return ["".join(c) for c in chunks]
def test_normalize_removes_accents():
assert normalize("Café Crème") == "cafe creme"
def test_words_drop_punctuation():
assert words("Hi, there!") == ["hi", "there"]
assert words("It's 9am.") == ["it's", "9am"]
def test_chunks_overlap():
assert joined(chunk(list("abcdefg"), 4, 1)) == ["abcd", "defg"]
def test_last_chunk_keeps_the_tail():
assert joined(chunk(list("abcdefgh"), 4, 1)) == ["abcd", "defg", "gh"]
def test_short_document_is_one_chunk():
assert joined(chunk(list("ab"), 4, 1)) == ["ab"]
def test_empty_text_has_no_chunks():
assert chunk_document("doc", "") == []
def test_rejects_bad_settings():
for size, overlap in [(0, 0), (3, 3), (3, 5), (3, -1)]:
with pytest.raises(ValueError):
chunk(list("abcdef"), size, overlap)
def test_chunks_carry_ids_and_normalized_text():
doc = chunk_document("faq", "Où est la gare?", size=3, overlap=1)
assert [text for _, text in doc] == ["ou est la", "la gare"]
assert [chunk_id for chunk_id, _ in doc] == ["faq:0", "faq:1"]
# Search Chunks
The help center's search embeds each article in chunks of about 200 words. chunk_document (chunks.py) splits an article into overlapping windows of words, so a sentence that crosses a boundary is still found whole in one chunk.
## Rules
- Text is lowercased and accents are removed before indexing, so "Café" and "cafe" match.
- Words are runs of letters, digits, and apostrophes. Punctuation is dropped.
- Each chunk holds size words and starts overlap words before the previous chunk ends. Every word lands in at least one chunk, so the last chunk may be shorter. A document shorter than size is one chunk, and an empty one has none.
- size is at least 1, and overlap is at least 0 and less than size.
- Chunk ids are the document id, a colon, and the chunk's position.
The tests are in test_chunks.py.
Test Output
No runs yet.