We are building a Network Incident Triage Agent that reads raw syslog snippets and customer trouble tickets, then returns structured severity scores, root-cause tags, and draft technician instructions. NOC teams at regional carriers and ISPs deal with thousands of unstructured alerts daily. This agent standardizes the first line of analysis so human engineers spend time fixing problems instead of reformatting tickets.
What you'll need
You need Python 3.10 or newer, the OpenAI SDK installed with pip install openai, and an active API key from Oxlo.ai. Sign up at https://portal.oxlo.ai, copy your key, and export it as OXLO_API_KEY in your shell. Oxlo.ai uses flat per-request pricing (see https://oxlo.ai/pricing), which means a verbose BGP debug log costs the same as a short SMS alert. That predictability matters when you are batch-processing overnight dumps or feeding long device configurations into the context window.
Step 1: Set up the Oxlo.ai client
I keep provider configuration isolated in a single module so I can switch keys or endpoints without touching business logic. Oxlo.ai is fully OpenAI-compatible, so the client initialization is identical to what you already know, just pointing at https://api.oxlo.ai/v1.
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)Step 2: Define the system prompt and schema
The system prompt is the contract. It forces the model to emit only valid JSON with a fixed schema, which lets downstream scripts route tickets without brittle regex. I also lock the severity vocabulary so the LLM cannot invent its own priority levels.
SYSTEM_PROMPT = """You are a senior telecom NOC engineer. Analyze the provided network incident text and output strictly valid JSON with these keys:
- severity: one of P1-Critical, P2-High, P3-Medium, P4-Low
- category: one of Backbone, Last-Mile, RF-Interference, Power, Configuration, Unknown
- probable_cause: a concise hypothesis of ten to twenty words
- recommended_action: one to two sentences of technician instruction
- affected_service: the customer-facing service name if identifiable, otherwise "Unspecified"
Do not include markdown formatting or explanation outside the JSON object."""Step 3: Create the triage function
The triage function is where the raw text meets the model. I use Llama 3.3 70B because it follows structured instructions reliably and runs without cold starts on Oxlo.ai. I set temperature low and enable JSON mode to reduce hallucinated keys. The function returns a native Python dict, ready for if-statements or database inserts.
import json
def triage_incident(raw_text: str) -> dict:
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": raw_text},
],
response_format={"type": "json_object"},
temperature=0.2,
)
content = response.choices[0].message.content
return json.loads(content)Step 4: Process a batch of incidents
NOC queues never arrive one at a time. This loop ingests a batch of alerts, triages each through Oxlo.ai, and immediately surfaces any P1-Critical item to stdout. Because Oxlo.ai charges per request rather than per token, you can feed multi-line syslogs or entire ticket threads into each call without watching the meter run. In production I replace that print statement with a webhook fire, but the skeleton stays the same. Wrapping each call in a try block prevents one malformed log from killing the entire batch.
incidents = [
"2024-05-21 03:14:22 BRT-CR-01 bgp[774]: neighbor 203.0.113.5 down, AS_PATH flap detected, 12k prefixes withdrawn",
"Ticket #8842: Enterprise customer reports 200 Mbps circuit dropping to 2 Mbps during peak hours for 3 days.",
"Site ID MN-TX-440: Generator fault LED active, battery bank at 11.2V, estimated runtime 18 minutes.",
]
results = []
for text in incidents:
try:
result = triage_incident(text)
results.append(result)
if result.get("severity") == "P1-Critical":
print(f"ESCALATE: {result}")
except Exception as e:
print(f"Parse failed for incident: {e}")Step 5: Generate technician dispatch notes
Structured JSON is perfect for automation, but field technicians need plain language. I pass the structured result to Kimi K2.6 to generate a concise dispatch note. Kimi handles the reasoning step of turning categories and causes into actionable sentences without requiring a separate template engine. The two-stage design keeps the first model focused on rigid classification and the second on human-readable summarization. You could skip this stage if you are feeding straight into a roboticket system, but for mixed human and automated workflows it cuts down on back-and-forth clarifications.
def generate_dispatch_note(structured: dict) -> str:
prompt = f"Turn this structured NOC output into a 2-sentence technician dispatch note. Be direct. JSON: {json.dumps(structured)}"
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": "You write terse field-technician dispatches."},
{"role": "user", "content": prompt},
],
temperature=0.3,
max_tokens=150,
)
return response.choices[0].message.content.strip()
for r in results:
note = generate_dispatch_note(r)
print(f"[{r['severity']}] {note}")Run it
Here is the complete script assembled from the steps above. Save it as triage.py, ensure your OXLO_API_KEY is exported, and run python triage.py. The output below shows what you should expect when the script hits Oxlo.ai with the sample incidents.
import os
import json
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ.get("OXLO_API_KEY")
)
SYSTEM_PROMPT = """You are a senior telecom NOC engineer. Analyze the provided network incident text and output strictly valid JSON with these keys:
- severity: one of P1-Critical, P2-High, P3-Medium, P4-Low
- category: one of Backbone, Last-Mile, RF-Interference, Power, Configuration, Unknown
- probable_cause: a concise hypothesis of ten to twenty words
- recommended_action: one to two sentences of technician instruction
- affected_service: the customer-facing service name if identifiable, otherwise "Unspecified"
Do not include markdown formatting or explanation outside the JSON object."""
def triage_incident(raw_text: str) -> dict:
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": raw_text},
],
response_format={"type": "json_object"},
temperature=0.2,
)
return json.loads(response.choices[0].message.content)
def generate_dispatch_note(structured: dict) -> str:
prompt = f"Turn this structured NOC output into a 2-sentence technician dispatch note. Be direct. JSON: {json.dumps(structured)}"
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": "You write terse field-technician dispatches."},
{"role": "user", "content": prompt},
],
temperature=0.3,
max_tokens=150,
)
return response.choices[0].message.content.strip()
incidents = [
"2024-05-21 03:14:22 BRT-CR-01 bgp[774]: neighbor 203.0.113.5 down, AS_PATH flap detected, 12k prefixes withdrawn",
"Ticket #8842: Enterprise customer reports 200 Mbps circuit dropping to 2 Mbps during peak hours for 3 days.",
"Site ID MN-TX-440: Generator fault LED active, battery bank at 11.2V, estimated runtime 18 minutes.",
]
for text in incidents:
structured = triage_incident(text)
note = generate_dispatch_note(structured)
print(f"Severity: {structured['severity']}")
print(f"Category: {structured['category']}")
print(f"Cause: {structured['probable_cause']}")
print(f"Dispatch: {note}")
print("-" * 40)Example output:
Severity: P1-Critical
Category: Backbone
Cause: BGP peering session flapped causing large-scale prefix withdrawal.
Dispatch: Investigate BGP session on BRT-CR-01 with neighbor 203.0.113.5. Verify upstream provider status and prepare route dampening if instability persists.
----------------------------------------
Severity: P2-High
Category: Last-Mile
Cause: Enterprise circuit experiencing severe throughput degradation during congestion windows.
Dispatch: Schedule field test of last-mile fiber for ticket 8842. Check optical levels and policer configuration before 6 PM peak window.
----------------------------------------
Severity: P1-Critical
Category: Power
Cause: Site generator fault with battery voltage critically low and short remaining runtime.
Dispatch: Dispatch technician to MN-TX-440 immediately with replacement battery and generator parts. Prioritize restore before 18-minute runtime expires.
----------------------------------------Wrap-up and next steps
This pipeline gives you a reproducible first-pass triage layer that turns noise into structured data. Two concrete next steps are wiring the P1 detector to an Opsgenie or PagerDuty webhook so critical events bypass email entirely, and adding a retrieval step over past incident embeddings using Oxlo.ai's embeddings endpoint to surface similar historical resolutions before the technician arrives on site.
