Use varnish+nginx+lua to build a website downgrading system

foreword

Usually after a website database hangs up, the consequences will be very serious. Basically the entire site is basically unusable. For some websites, it would be nice if they could provide basic browsing services when the database is down. This article will try to use varnish + nginx + lua to build a website downgrade system to achieve the whole goal.

relegation target

The goal of the downgrade solution is to display the cached page data to the user when a fatal failure occurs on the website (for example, a 500 error occurs and the service cannot be provided). This provides basic browsing services.
1. Only provide basic browsing services
2. The browsed data is all data in the non-login state
3. Support manual and automatic downgrade. Automatic downgrade is when the backend returns 500 errors and reaches a certain threshold within a period of time (excluding 503). Manual downgrades are performed from the control interface.

downgrade plan

storage

Use varnish as storage. It effectively saves physical memory and maintains good performance.

renew

Use the crond script to analyze the request URL from the nginx access log, and then send a request to Varnish to update Varnish’s cache. The asynchronous update of the cache reduces the pressure on nginx.

downgrade

Support manual downgrade and automatic downgrade. After downgrading, nginx automatically extracts data from varnish and returns it to the user.

flow chart

Process description

1. When a user requests nginx, nginx will judge whether it is currently in a degraded state. If it is a degraded state, get the data directly from Varnish. Non-degraded state, forward the request to php-fpm.
2. When the crond script requests Varnish to update the cache data, if the current Varnish is in a degraded state, the cache update will not be performed. If it is not in a degraded state, transfer the request to nginx to obtain data. Then cache the fetched data in Varnish.
3. Varnish will automatically monitor the status of the backend nginx. If it detects that nginx is already in a downgraded state, Varnish will automatically enter into a downgraded state as well.

Installation and deployment

<p style="font-size:14px;vertical-align:baseline;color:#444444;font-family:'Microsoft Yahei', 'Helvetica Neue', Helvetica, Arial, sans-serif;background-color:#FFFFFFxy_pass http://varnish$str_params;

}

location ~* ^(.+\.php)(.*)$ {

#check downgrade status, then get data from varnish

set $str_params $uri;

content_by_lua_file lua/pc_get_downgrade_data.lua;

}

location /hl_get_auto_status {

if ($white_ip = 0) {

return 403;

}

  content_by_lua_file lua/pc_get_auto_status.lua;

}

location /hl_get_status {

if ($white_ip = 0) {

return 403;

}

content_by_lua_file lua/pc_get_status.lua;

}

location /hl_set_status {

if ($white_ip = 0) {

return 403;

}

  content_by_lua_file lua/pc_set_status.lua;

}

the

log_by_lua_file lua/pc_status_stat.lua;

Deploy cron script

The script varnish_crond.php. Add execution command in crond. Executes every minute.
Request from crond, user-agent data is varnish_crond. Set user-agent as varnish_crond to request special handling. Guaranteed to request normally and return relevant data.

Downgrade management

varnish downgrade

Just make the monitoring script check.php specified in the varnish configuration return a 500 error. Varnish monitors that the specified script is unavailable, and automatically enters the degraded state.
When the script returns to a 200 status, Varnish automatically returns to normal.

downgrade nginx

set downgrade

Back to normal

View downgrade status

If the returned value is 1, it means downgrade

View downgrade status

If the returned value is 1, it means downgrade