We are going to build a primary-source analyzer that ingests a raw historical text and returns structured JSON covering historical context, key figures, sentiment, and potential historiographical bias. The goal is to give history and literature scholars a reproducible first pass on archival documents before they commit to months of close reading. The entire tool runs in under a hundred lines of Python against the Oxlo.ai API, so you can process a box of letters with the same rigor you apply to a single page.
What you'll need
You will need Python 3.10 or newer so you can use modern pathlib and typing syntax without compatibility shims. Install the OpenAI SDK with pip install openai, then grab an Oxlo.ai API key from https://portal.oxlo.ai. I recommend storing the key in an environment variable named OXLO_API_KEY so it never touches disk inside your script. I use the standard openai package rather than a custom HTTP client because Oxlo.ai is fully OpenAI SDK compatible, which means these snippets work in Python and Node.js with only surface-level syntax changes. Oxlo.ai offers a free tier that includes 60 requests per day across more than sixteen models, which is enough to prototype this pipeline on a small test set before you move to a paid plan. Because Oxlo.ai uses flat request-based pricing, a 4,000-word sermon costs exactly the same as a one-paragraph telegram, which keeps grant budgets predictable when you are working with uneven archival material.
Step 1: Configure the Oxlo.ai client
Instantiate the OpenAI-compatible client pointing at Oxlo.ai and verify connectivity with llama-3.3-70b, a general-purpose model that handles long-form historical prose and nineteenth-century syntax reliably. Since Oxlo.ai serves popular models with no cold starts, the first request of the day returns just as fast as the hundredth, which matters when you are iterating on a prompt between archive visits.
from openai import OpenAI
import os
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ.get("OXLO_API_KEY")
)
# Verify the endpoint is alive
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[{"role": "user", "content": "Say 'API is live'"}],
)
print(response.choices[0].message.content)
Step 2: Define the system prompt
The system prompt below forces the model to act as a careful archival analyst and to return only a predictable JSON object with predetermined keys, which keeps output consistent across hundreds of documents. You can edit the schema to match your own field, whether that is medieval charters, Victorian diaries, or postcolonial novels, and the model will still obey the structure.
SYSTEM_PROMPT = """You are an archival research assistant with expertise in historical and literary analysis.
Analyze the primary source provided by the user and return a single JSON object.
Do not include markdown formatting, explanations, or text outside the JSON.
Required keys:
- "summary": A concise 2-sentence summary of the document.
- "historical_context": One paragraph situating the document in its era.
- "key_figures": An array of objects, each with "name" and "role".
- "sentiment": The emotional tone of the author, e.g., "anxious", "triumphant", "neutral".
- "bias_indicators": An array of phrases or framing choices that reveal authorial bias.
- "cited_passages": An array of up to 3 exact short quotes that support your analysis.
Be rigorous. If the text is ambiguous, say so."""
Step 3: Build the core analysis function
This function sends the document text to Oxlo.ai and parses the JSON response, using a low temperature to reduce hallucinations on rare names and dates that often confuse less constrained models. If your archive is multilingual, swap the model string to qwen-3-32b without changing any other code, because Oxlo.ai exposes every model through the same base URL.
import json
def analyze_primary_source(text: str) -> dict:
"""Send text to Oxlo.ai and return parsed JSON analysis."""
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": text},
],
temperature=0.2,
)
raw = response.choices[0].message.content.strip()
# Defensive parsing in case the model wraps JSON in markdown fences
if raw.startswith("```json"):
raw = raw[7:]
if raw.endswith("```"):
raw = raw[:-3]
return json.loads(raw.strip())
Step 4: Ingest and clean text
Real archives are often messy OCR text files full of control characters and page headers, so this helper normalizes whitespace and removes null bytes before the text reaches the model. For very long typescripts or bound volumes, switch the model to kimi-k2.6 on Oxlo.ai to leverage its 131K context window without worrying about per-token cost escalation.
import re
from pathlib import Path
def load_document(path: str) -> str:
"""Read and clean a text file."""
text = Path(path).read_text(encoding="utf-8", errors="ignore")
# Collapse multiple whitespace and remove null bytes
text = re.sub(r"\s+", " ", text)
text = text.replace("\x00", "")
# Cap to roughly 12k words for llama-3.3-70b; increase if you change model
return text[:60000]
Step 5: Batch process a folder
Most research projects involve dozens of letters or newspaper clippings, so this snippet loops over every .txt file in a directory, runs the analyzer, and writes one JSON file per source. Because Oxlo.ai charges per request rather than per token, you can estimate the cost of processing the whole corpus upfront simply by counting files, which makes grant budgeting straightforward.
from pathlib import Path
SOURCES_DIR = Path("sources")
OUTPUT_DIR = Path("analysis")
OUTPUT_DIR.mkdir(exist_ok=True)
for txt_path in sorted(SOURCES_DIR.glob("*.txt")):
print(f"Processing {txt_path.name} ...")
raw_text = load_document(txt_path)
try:
result = analyze_primary_source(raw_text)
out_path = OUTPUT_DIR / f"{txt_path.stem}.json"
out_path.write_text(json.dumps(result, indent=2, ensure_ascii=False))
print(f" Wrote {out_path}")
except Exception as e:
print(f" Failed on {txt_path.name}: {e}")
Run it
Create a sample source file and execute the script from your terminal. I have included a short synthetic Civil War letter below so you can verify the pipeline end-to-end without digging through an archive. You can also point the script at a real archive folder immediately; just ensure every document has a .txt extension and UTF-8 encoding.
# Terminal
mkdir -p sources
echo "My dearest Margaret, The regiment marches south at dawn. I fear this campaign shall be longer than the newspapers admit. The men speak of Richmond not as a victory waiting, but as a grave. Yours, Thomas." > sources/letter_1863.txt
export OXLO_API_KEY="YOUR_OXLO_API_KEY"
python analyze.py
After the script finishes, open analysis/letter_1863.json. You should see structured fields that you can import directly into Python pandas, R, or a qualitative analysis tool like NVivo. If you instead get a JSONDecodeError, check that the source file is not empty and that no garbled UTF-8 sequence truncated the text mid-byte, because raw OCR dumps often contain invalid continuation bytes that confuse the parser.
{
"summary": "A Civil War soldier writes to his wife expressing private fears about an upcoming campaign near Richmond.",
"historical_context": "Written during the American Civil War, this letter reflects the growing skepticism among Union soldiers regarding rapid victory claims made by contemporary newspapers in 1863.",
"key_figures": [
{"name": "Thomas", "role": "Union soldier and author"},
{"name": "Margaret", "role": "Recipient and wife"}
],
"sentiment": "anxious",
"bias_indicators": [
"Contrasts newspaper optimism with soldiers' private dread",
"Frames Richmond as a grave rather than a prize"
],
"cited_passages": [
"I fear this campaign shall be longer than the newspapers admit.",
"The men speak of Richmond not as a victory waiting, but as a grave."
]
}
With the output verified, you can now drop an entire folder of .txt files into sources/ and rerun. The resulting JSON artifacts act as structured front matter for each document, making it possible to filter your collection by sentiment, bias type, or mentioned figures before you ever open the original file.
Wrap up and next steps
This pipeline gives you a reproducible, code-first way to pre-process archival collections without surrendering scholarly rigor to a black-box interface. Two concrete directions to take it next:
- Feed the resulting JSON into a vector database such as Chroma or Pinecone. You can then semantically search across your entire corpus for specific sentiments, biases, or figures without reading every file manually. This is especially powerful when you are working with hundreds of diary entries and need to isolate every passage that mentions economic anxiety or imperial framing.
- Wrap the analyzer in a lightweight Streamlit or Gradio interface so non-technical collaborators in your department can upload scans, tweak the system prompt, and download JSON results without touching Python. Because the backend is just the standard OpenAI client pointing at Oxlo.ai, your web layer requires no vendor-specific adapters.
Oxlo.ai fits this workflow well because its request-based pricing removes the penalty for long documents, and its OpenAI-compatible SDK means you can drop these exact snippets into existing academic code without rewriting your HTTP layer. For projects that outgrow a single machine, you could parallelize the loop with concurrent.futures and still hit the same Oxlo.ai endpoint, because the API handles concurrent requests without cold-start latency. If you want to see how the pricing model compares for high-volume archival work, or if you need dedicated GPUs for an entire research group, check https://oxlo.ai/pricing.
