Let the data written to the database be automatically written to the cache_weixin_34390996’s blog

In project development, in order to reduce the I/O pressure of the database and speed up the response speed of requests, caching is a commonly used technology. Redis and Memcache are two commonly used to do Data caching technology. Some common methods of data caching are to let the data be automatically synchronized to the cache through some automated scripts after being written to the database, or to manually write data to the cache once after writing data to the database. These practices are unavoidably cumbersome, and the code is not easy to maintain. When I was writing a Node.js project, I found that using Mongoose (a MongoDB ODM) and Sequelize (a MySQL ORM) has some features that can elegantly allow data written to MongoDB/MySQL to be automatically written to Redis, and when doing query operations, it can automatically look up data from the cache first. If it cannot be found in the cache, it will enter the DB to search, and write the data found in the DB to the cache.

This article does not explain the basic usage of Mongoose and Sequelize, here only explains how to achieve the automatic caching mentioned above.

Some libraries used in this article are Mongoose, Sequelize, ioredis and lodash. Node.js version is 7.7.1.

Automatic caching in MongoDB

// redis.js
 const Redis = require('ioredis');
 const Config = require('../config');

 const redis = new Redis(Config.redis);

 module.exports = redis;

The code in the above file is mainly used to connect to redis.

// mongodb.js
 const mongoose = require('mongoose');

 mongoose. Promise = global. Promise;
 const demoDB = mongoose.createConnection('mongodb://127.0.0.1/demo', {});

 module.exports = demoDB;

The above is the code to connect to mongodb.

// mongoBase.js
 const mongoose = require('mongoose');
 const Schema = mongoose.Schema;
 const redis = require('./redis');

 function baseFind(method, params, time) {
   const self = this;
   const collectionName = this. collection. name;
   const dbName = this.db.name;
   const redisKey = [dbName, collectionName, JSON.stringify(params)].join(':');
   const expireTime = time || 3600;

   return new Promise(function (resolve, reject) {
     redis.get(redisKey, function (err, data) {
       if (err) return reject(err);
       if (data) return resolve(JSON. parse(data));

       self[method](params).lean().exec(function (err, data) {
         if (err) return reject(err);
         if (Object. keys(data). length === 0) return resolve(data);

         redis.setex(redisKey, expireTime, JSON.stringify(data));
         resolve(data);
       });
     });
   });
 }

 const Methods = {
   findCache(params, time) {
     return baseFind. call(this, 'find', params, time);
   },
   findOneCache(params, time) {
     return baseFind. call(this, 'findOne', params, time);
   },
   findByIdCache(params, time) {
     return baseFind. call(this, 'findById', params, time);
   },
 };

 const BaseSchema = function () {
   this.defaultOpts = {
   };
 };

 BaseSchema.prototype.extend = function (schemaOpts) {
   const schema = this. wrapMethods(new Schema(schemaOpts, {
     toObject: { virtuals: true },
     toJSON: { virtuals: true },
   }));

   return schema;
 };

 BaseSchema.prototype.wrapMethods = function (schema) {
   schema. post('save', function (data) {
     const dbName = data.db.name;
     const collectionName = data. collection. name;
     const redisKey = [dbName, collectionName, JSON.stringify(data._id)].join(':');

     redis.setex(redisKey, 3600, JSON.stringify(this));
   });

   Object.keys(Methods).forEach(function (method) {
     schema.statics[method] = Methods[method];
   });
   return schema;
 };

 module.exports = new BaseSchema();

When the above code is modeled with mongoose, all schemas will inherit this BaseSchema. This BaseSchema adds a document middleware that is triggered after the save method is executed for all schemas that inherit it. The function of this middleware is to automatically write redis after the data is written to MongoDB. Then three static methods are added to each schema that inherits it, namely findByIdCache, findOneCache and findCache, which are extensions of findById, findOne and find methods, but the difference is that when querying with the three added methods It will first search for data from redis according to the incoming conditions, and return the data if it is found, and continue to call the corresponding native method to search in MongoDB if it is not found, and if it is found in MongoDB, it will be found. The data is written to redis for future queries. The calling methods of the three added static methods are consistent with their corresponding native methods, except that one more time can be passed in to set the data in the cache.Expiration.

// userModel.js
 const BaseSchema = require('./mongoBase');
 const mongoDB = require('./mongodb.js');

 const userSchema = BaseSchema. extend({
   name: String,
   age: Number,
   addr: String,
 });

 module.exports = mongoDB.model('User', userSchema, 'user');

This is a model built for the user collection, which inherits the middleware and static methods mentioned above through the BaseSchema.extend method.

// index.js
 const UserModel = require('./userModl');

 const action = async function () {
   const user = await UserModel.create({ name: 'node', age: 7, addr: 'nodejs.org' });
   const data1 = await UserModel.findByIdCache(user._id.toString());
   const data2 = await UserModel. findOneCache({ age: 7 });
   const data3 = await UserModel. findCache({ name: 'node', age: 7 }, 7200);
   return [data1, data2, data3];
 };

 action().then(console.log).catch(console.error);

The above code is to write a piece of data to the User collection, and then call the three static methods we added for query in turn. Open the monitor of redis and find that the code has been executed as we expected.

To sum up, the above solution mainly achieves the functions we want through Mongoose’s middleware and static methods. However, the added findOneCache and findCache methods are difficult to achieve high data consistency. If you want to pursue strong data consistency, use their corresponding findOne and find. findByIdCache can guarantee good data consistency, but it is limited to querying and saving when modifying data. If it is directly updated, data consistency cannot be achieved.

Implementing automatic caching in MySQL

// mysql.js
 const Sequelize = require('sequelize');
 const _ = require('lodash');
 const redis = require('./redis');

 const setCache = function (data) {
   if (_.isEmpty(data) || !data.id) return;

   const dbName = data.$modelOptions.sequelize.config.database;
   const tableName = data. $modelOptions. tableName;
   const redisKey = [dbName, tableName, JSON. stringify(data. id)]. join(':')
   redis.setex(redisKey, 3600, JSON.stringify(data.toJSON()));
 };

 const sequelize = new Sequelize('demo', 'root', '', {
   host: 'localhost',
   port: 3306,
   hooks: {
     afterUpdate(data) {
       setCache(data);
     },
     afterCreate(data) {
       setCache(data);
     },
   },
 });

 sequelize
   .authenticate()
   .then(function () {
     console.log('Connection has been established successfully.');
   })
   .catch(function (err) {
     console.error('Unable to connect to the database:', err);
   });

 module.exports = sequelize;

The main function of the above code is to connect to MySQL and generate a sequelize instance. When building a sequelize instance, two hook methods afterUpdate and afterCreate are added. afterUpdate is used to execute the function after the model instance is updated. Note that the model instance must be updated to trigger this method. If the update is directly similar to Model.update, this hook function will not be triggered , this hook can only be triggered when an existing instance calls the save method. afterCreate is a hook function called after the model instance is created. The main purpose of these two hooks is to automatically write to redis after a piece of data is written to MySQL, that is, to realize automatic caching.

// mysqlBase.js
 const _ = require('lodash');
 const Sequelize = require('sequelize');
 const redis = require('./redis');

 function baseFind(method, params, time) {
   const self = this;
   const dbName = this.sequelize.config.database;
   const tableName = this.name;
   const redisKey = [dbName, tableName, JSON. stringify(params)]. join(':');
   return (async function () {
     const cacheData = await redis. get(redisKey);
     if (!_.isEmpty(cacheData)) return JSON.parse(cacheData);

     const dbData = await self[method](params);
     if (_.isEmpty(dbData)) return {};

     redis.setex(redisKey, time || 3600, JSON.stringify(dbData));
     return dbData;
   })();
 }

 const Base = function (sequelize) {
   this. sequelize = sequelize;
 };

 Base.prototype.define = function (model, attributes, options) {
   const self = this;
   return this.sequelize.define(model, _.assign({
     id: {
       type: Sequelize. UUID,
       primaryKey: true,
       defaultValue: Sequelize.UUIDV1,
     },
   }, attributes), _.defaultsDeep({
     classMethods: {
       findByIdCache(params, time) {
         this. sequelize = self. sequelize;
         return baseFind. call(this, 'findById', params, time);
       },
       findOneCache(params, time) {
         this. sequelize = self. sequelize;
         return baseFind. call(this, 'findOne', params, time);
       },
       findAllCache(params, time) {
         this. sequelize = self. sequelize;
         return baseFind. call(this, 'findAll', params, time);
       },
     },
   }, options));
 };

 module.exports = Base;

The above code has roughly the same effect as the previous mongoBase. When modeling in sequelize, all schemas will inherit this Base. This Base adds three static methods for all schemas that inherit it, namely findByIdCache, findOneCache and findAllCache. Their functions are the same as those of the previous three methods in mongoBase, just to be consistent with the original findAll in sequelize. , findCache becomes findAllCache here. Adding class methods (classMethods) to schema in sequelize is equivalent to adding static methods (statics) to schema in mongoose.

// mysqlUser.js
 const Sequelize = require('sequelize');
 const base = require('./mysqlBase.js');
 const sequelize = require('./mysql.js');

 const Base = new base(sequelize);
 module.exports = Base.define('user', {
   name: Sequelize. STRING,
   age: Sequelize. INTEGER,
   addr: Sequelize. STRING,
 }, {
   tableName: 'user',
   timestamps: true,
 });

A User schema is defined above, which inherits findByIdCache, findOneCache and findAllCache from Base.

const UserModel = require('./mysqlUser');

 const action = async function () {
   await UserModel.sync({ force: true });
   const user = await UserModel.create({ name: 'node', age: 7, addr: 'nodejs.org' });
   await UserModel.findByIdCache(user.id);
   await UserModel. findOneCache({ where: { age: 7 }});
   await UserModel. findAllCache({ where: { name: 'node', age: 7 }}, 7200);
   return 'finish';
 };

 action().then(console.log).catch(console.error);

In summary, the automatic caching scheme implemented by sequelize is the same as that implemented by mongoose before, and there will also be data consistency problems. findByIdCache is better, findOneCache and findAllCache are worse. Of course, many details here are not considered perfect, and can be based on business rationality Adjustment.

Base. This Base adds three static methods for all schemas that inherit it, namely findByIdCache, findOneCache and findAllCache. Their functions are the same as those of the previous three methods in mongoBase, just to be consistent with the original findAll in sequelize. , findCache becomes findAllCache here. Adding class methods (classMethods) to schema in sequelize is equivalent to adding static methods (statics) to schema in mongoose.

// mysqlUser.js
 const Sequelize = require('sequelize');
 const base = require('./mysqlBase.js');
 const sequelize = require('./mysql.js');

 const Base = new base(sequelize);
 module.exports = Base.define('user', {
   name: Sequelize. STRING,
   age: Sequelize. INTEGER,
   addr: Sequelize. STRING,
 }, {
   tableName: 'user',
   timestamps: true,
 });

A User schema is defined above, which inherits findByIdCache, findOneCache and findAllCache from Base.

const UserModel = require('./mysqlUser');

 const action = async function () {
   await UserModel.sync({ force: true });
   const user = await UserModel.create({ name: 'node', age: 7, addr: 'nodejs.org' });
   await UserModel.findByIdCache(user.id);
   await UserModel. findOneCache({ where: { age: 7 }});
   await UserModel. findAllCache({ where: { name: 'node', age: 7 }}, 7200);
   return 'finish';
 };

 action().then(console.log).catch(console.error);

In summary, the automatic caching scheme implemented by sequelize is the same as that implemented by mongoose before, and there will also be data consistency problems. findByIdCache is better, findOneCache and findAllCache are worse. Of course, many details here are not considered perfect, and can be based on business rationality Adjustment.