Improved analysis of LUG file-related bulk data using LLG

Nowadays, organizations generate a significant volume of log files that require processing for condition checking, debugging, and anomaly resolution. Outsourcing such processing is not suitable due to the need for real-time processing and security maintenance. Given the multitude of different software and services, organizations face a substantial volume of production logs that should be processed rather than deleted or ignored. In the traditional approach, experts manually check the logs daily. This, on one hand, slows down the process, increases the time and inaccuracy, and, on the other hand, results in a high hiring cost due to the need for an expert force. This article introduces a solution that employs generative neural networks to establish a local structure for log analysis within the organization. The process involves retrieving and parsing text files from various sectors, segmenting them into manageable portions, embedding them, and storing them in a vector database. In this structure, a trained individual without special expertise can quickly access necessary information using appropriate prompts from a local language model available at any time. Therefore, the proposed method can increase the stability of security, increase the speed of analysis, and reduce the costs of human resources.

Keywords:

neural network , generative artificial intelligence , large language model , LLM , Log File.