Web Log Collection Monitoring System by using ELK

Abstract

Establishment and focus: As internet becomes widely spread, users of web and apps are increasing, and even though specification of servers and the network availability are being upgraded to handle a large number of user sessions and traffics in web servers, there are many limitations in solving the problems. Current situation is that the number of servers gets increased to distribute requests of users in order to solve the problem of insufficient capacity to handle sessions and traffics as an alternative solution. Users are requesting the processing of various functions in the webpages, and since all these processes are recorded in the web logs, the amount of the logs is parallel to the number of requests by the users. These logs are frequently used to identify the causes when web issues occur. Since the service administrator has to check logs of each server by accessing a large number of servers one by one in such procedure, it consumes a lot of time resources to check the web logs to identify the web issues.
System: System: In order to reduce the resource, various methods such as FTP (File Transfer Protocol) and Filebeat are used to collect logs of a large number of servers into the central server. In case of using FTP, a user can roll and send logs by designating certain time period or date, and in case of Filebeat, events can be sent through event detection in real-time. But if there is a server of which the transmission is omitted in the process of collecting logs, the reliability of the log analysis data has to drop accordingly. In this study, a plan to enhance the reliability of log analysis data through a system that can monitor whether there is any server of which the collection has been omitted by using the access log analysis data of Apache web server using ELK is proposed. By using a model that is proposed in this study, the time resources that are consumed to collect logs can be saved. It is expected to provide smooth service to users by reducing the time to identify the cause and take necessary measures since integrated logs are checked when the log data needs to be checked due to the occurrence of a failure in future. As for the future study, the study that can establish the process which can automatically take necessary actions through the analysis of the logs. That are recorded during the occurrence of failures shall be continued.