Using LLMs and ESRE to find similar user sessions

This article reports experiments using GPT-4 to categorize 75 Linux user sessions into nine behavior-based categories and applies Elastic’s ELSER for semantic search over generated summaries. The team found that adding category examples (few-shot) improved classification accuracy by ~20% and noted challenges with multi-label ambiguity and confusion between similar categories (e.g., Linux Command Line Utility vs. Process Execution). For semantic search, ELSER outperformed BM25 on conceptual queries but was sensitive to prompt-injected keywords (e.g., ubiquitous use of “malicious”), with difficulty distinguishing certain nuances (interactive vs. non-interactive). Future work targets improving summarization with RAG via ESRE.