Using LLMs to summarize user sessions

Elastic’s Security ML team reports lessons learned from using GPT-4 to summarize Linux user sessions, finding that aggregated JSON snapshots, high-level few-shot prompts, and low temperature settings improve consistency while occasional hallucinations persist. The team evaluated outputs via human review and automated metrics (ROUGE, BLEU, METEOR, BERTScore, BLANC), noting better alignment with human judgments than with reference-based metrics. Future work includes applying retrieval-augmented generation (RAG) with ESRE and exploring automated session categorization.