Nginxlog real-time monitoring system based on Storm

Background UAE (UC App Engine) is a PaaS platform inside UC. The overall structure is somewhat similar to CloudFoundry, including: Rapid deployment: support Node.js, Play!, PHP and other frameworks Information transparency: operation and maintenance process, system status, business status Gray scale trial and error: IP gray scale, regional gray scale Basic services: key-value storage, MySQL high availability, picture platform, etc. It is not the main character here and will not be introduced in detail. There are hundreds of web applications running on the UAE, and all requests will be routed through the UAE. The daily Nginx access log size is terabytes. How to monitor the access trends of each business, advertising data, page time consumption, access quality, Custom reports and exception alarms? Hadoop can meet the statistical requirements, but the second-level real-time performance is not enough; using Spark Streaming is a bit overkill, and we have no Spark engineering experience; self-written distributed program scheduling is cumbersome and needs to consider expansion and message flow; Finally, our technology selection is Storm: relatively lightweight, flexible, convenient for message transmission, and flexible in expansion. In addition, since there are many clusters in UC, cross-cluster log transmission will also be a relatively big problem.…