Redis

Redis is an open-source, networked, in-memory, key-value data store with optional durability, written in ANSI C.

Setup

Cache

If you have not already, you'll need to configure a Redis instance and install a Redis client library for PHP. Most environments require the phpredis PHP extension. On Debian / Ubuntu, you can install the requirements with the following command:

$ apt-get install redis-server php-redis

In your "LocalSettings.php" file, set $wgObjectCaches:

$wgObjectCaches['redis'] = [
    'class'                => 'RedisBagOStuff',
    'servers'              => [ '127.0.0.1:6379' ],
    // 'connectTimeout'    => 1,
    // 'persistent'        => false,
    // 'password'          => 'secret',
    // 'automaticFailOver' => true,
];

Parameters explained

Name	Required?	Description	Default
`servers`	Required	An array of server names. A server name may be a hostname, a hostname/port combination or the absolute path of a UNIX socket. If a hostname is specified but no port, the standard port number 6379 will be used. Arrays keys can be used to specify the tag to hash on in place of the host/port.
`connectTimeout`	Optional	The timeout for new connections, in seconds.	1 second
`persistent`	Optional	Set this to true to allow connections to persist across multiple web requests.	false
`password`	Optional	The authentication password, will be sent to Redis in clear text. If it is unspecified, no AUTH command will be sent.
`automaticFailover`	Optional	If this is false, then each key will be mapped to a single server, and if that server is down, any requests for that key will fail. If this is true, a connection failure will cause the client to immediately try the next server in the list (as determined by a consistent hashing algorithm). This has the potential to create consistency issues if a server is slow enough to flap, for example if it is in swap death.	true

You will now be able to acquire a Redis object cache object via ObjectCache::getInstance( 'redis' ). If you'd like to use Redis as the default cache for various data, you may set any of the following configuration options:

$wgMainCacheType = 'redis';

If using redis as a cache, make sure to configure redis with an appropriate maxmemory and maxmemory-policy setting

Job queue

Make sure the current job queue is empty before cutover, use showJobs.php to check the backlog. Drain the queue with runJobs.php before declaring a short read-only maintenance session, if needed.

$wgJobTypeConf['default'] = [
    'class'          => 'JobQueueRedis',
    'redisServer'    => '127.0.0.1:6379',
    'redisConfig'    => [],
    'daemonized'     => true
 ];

Parameters explained

redisConfig: An array of parameters to RedisConnectionPool::__construct(). Note that the serializer option is ignored as "none" is always used. If the same Redis server is used as for $wgObjectCaches, the Redis password needs to be set here as well (see $wgObjectCaches config above).
redisServer: A hostname/port combination or the absolute path of a UNIX socket. If a hostname is specified but no port, the standard port number 6379 will be used. Required.
compression: The type of compression to use; one of (none,gzip).
daemonized: Currently it doesn't support setting it to false.

From that moment, jobs will be delivered to the Redis instance on the specified server.

Follow the next section to configure redisJobChronService and make sure it's running continuously. Otherwise, delayed jobs would never get undelayed and executed.

Automatic handling of job recycling and abandons

Unlike the default database-based job queues, MediaWiki do not actively manage the lifecycle of abandoned or failed jobs for Redis-based queues, and some extensions (including CirrusSearch) would use delayed jobs for performance if the job queue is configured to use Redis. To purge abandoned jobs from the queue and reschedule failed or delayed jobs, you need an additional job runner service.

Clone the git repository https://github.com/wikimedia/mediawiki-services-jobrunner

Create a configuration file named config.json:

{
	"groups": {
		"basic": {
			"runners": 0
		}
	},
	"limits": {
	},
	"redis": {
		"aggregators": [
			"127.0.0.1:6379"
		],
		"queues": [
			"127.0.0.1:6379"
		]
	},
	"dispatcher": "nothing"
}

Configure a daemon to run this at server start:

$ php redisJobChronService --config-file=config.json

The minimal config above is only applicable to redisJobChronService, if you want to set $wgJobRunRate to 0 or drain the queue faster while idle, you will still have to configure your job queue service. Preferably, run redisJobRunnerService from the same repo with dispatcher configured and the number of runners appropriate to your system. The dispatcher for a normal setup can be simply runJobs.php with some limit parameters:

php /path/to/maintenance/run.php runJobs --wiki=%(db)x --type=%(type)x --maxtime=%(maxtime)x --maxjobs=20 --memory-limit=%(maxmem)x --result=json

The %(...)x syntax should be kept as-is for the service to fill-in in runtime, and --maxjobs=20 is to make sure the batch of jobs finish before timeout (maxtime). Lower the value if the runner service complains about partial JSON output.

The example config from the repo explained the available config fields briefly, you may start simple with only a wildcard group and only add additional groups based on the observed workload pattern if needed.

MediaWiki & Wikimedia use cases for Redis

History of job queue runners at WMF on Wikitech.

Redis

Setup

Cache

Job queue

Automatic handling of job recycling and abandons

MediaWiki & Wikimedia use cases for Redis

Further reading

General

Analytics

Tooling

Informed Opinions

Miscellaneous