redux provides a full interface to the Redis API; it
provides a hiredis driver wrapped in R and uses this to expose all 199
Redis commands as a set of user-friendly R functions that do basic error
checking.
It is possible to build user-friendly applications on top of this,
for example storr which
provides a content-addressable object store, and rrqueue /
rrq which
implement a scalable queuing system.
The main entry point for creating a redis_api object is
the hiredis function:
r <- redux::hiredis()By default, it will connect to a database running on the local
machine (127.0.0.1) and port 6379. To connect to a
different host, or to specify a password, initial database, or to use a
socket connection, use the command .
The redis_api object is an R6 class with
many methods, each corresponding to a different Redis
command.
r## <redis_api>
## Redis commands:
## APPEND: function
## AUTH: function
## BGREWRITEAOF: function
## BGSAVE: function
## ...
## ZSCORE: function
## ZUNIONSTORE: function
## Other public methods:
## clone: function
## command: function
## config: function
## initialize: function
## pipeline: function
## reconnect: function
## subscribe: function
## type: function
## version: functionFor example, SET and GET:
r$SET("mykey", "mydata") # set the key "mykey" to the value "mydata"## [Redis: OK]
r$GET("mykey")## [1] "mydata"The value for most arguments must be a string or will be coerced into
one; clearly this is not going to be suitable for most R objects. The
solution is to serialise the R object. redux can
accept objects serialised to strings or to byte streams, and the
functions the object_to_bin and
object_to_string functions can help here, serialising the
objects to binary and string representations. (Alternatively you can do
this yourself using serialize.)
obj <- redux::object_to_bin(1:10)
obj## [1] 42 0a 03 00 00 00 02 03 04 00 00 05 03 00 05 00 00 00 55 54 46 2d 38 ee 00
## [26] 00 00 02 00 00 00 01 00 00 00 09 00 04 00 0e 00 00 00 63 6f 6d 70 61 63 74
## [51] 5f 69 6e 74 73 65 71 02 00 00 00 01 00 00 00 09 00 04 00 04 00 00 00 62 61
## [76] 73 65 02 00 00 00 0d 00 00 00 01 00 00 00 0d 00 00 00 fe 00 00 00 0e 00 00
## [101] 00 03 00 00 00 00 00 00 00 00 00 24 40 00 00 00 00 00 00 f0 3f 00 00 00 00
## [126] 00 00 f0 3f fe 00 00 00or
str <- redux::object_to_string(1:10)
str## [1] "A\n3\n262914\n197888\n5\nUTF-8\n238\n2\n1\n262153\n14\ncompact_intseq\n2\n1\n262153\n4\nbase\n2\n13\n1\n13\n254\n14\n3\n0x1.4p+3\n0x1p+0\n0x1p+0\n254\n"The binary serialisation is faster, smaller, and preserves all the
bits of floating point numbers. The string version might be preferable
where having only strings in the database is wanted. The binary
serialisation is compatible with the same approach used in
RcppRedis, though it is never done automatically.
These values can be deserialised:
redux::bin_to_object(obj)## [1] 1 2 3 4 5 6 7 8 9 10
redux::string_to_object(str)## [1] 1 2 3 4 5 6 7 8 9 10So:
r$SET("mylist", redux::object_to_bin(1:10))## [Redis: OK]
r$GET("mylist")## [1] 42 0a 03 00 00 00 02 03 04 00 00 05 03 00 05 00 00 00 55 54 46 2d 38 ee 00
## [26] 00 00 02 00 00 00 01 00 00 00 09 00 04 00 0e 00 00 00 63 6f 6d 70 61 63 74
## [51] 5f 69 6e 74 73 65 71 02 00 00 00 01 00 00 00 09 00 04 00 04 00 00 00 62 61
## [76] 73 65 02 00 00 00 0d 00 00 00 01 00 00 00 0d 00 00 00 fe 00 00 00 0e 00 00
## [101] 00 03 00 00 00 00 00 00 00 00 00 24 40 00 00 00 00 00 00 f0 3f 00 00 00 00
## [126] 00 00 f0 3f fe 00 00 00
redux::bin_to_object(r$GET("mylist"))## [1] 1 2 3 4 5 6 7 8 9 10Using string serialisation is similar:
r$SET("mylist", redux::object_to_string(1:10))## [Redis: OK]
r$GET("mylist")## [1] "A\n3\n262914\n197888\n5\nUTF-8\n238\n2\n1\n262153\n14\ncompact_intseq\n2\n1\n262153\n4\nbase\n2\n13\n1\n13\n254\n14\n3\n0x1.4p+3\n0x1p+0\n0x1p+0\n254\n"
redux::string_to_object(r$GET("mylist"))## [1] 1 2 3 4 5 6 7 8 9 10This gives you all the power of Redis, but you will have to manually serialise/deserialise all complicated R objects (i.e., everything other than logicals, numbers or strings). Similarly, you are responsible for type coercion/deserialisation when retrieving data at the other end.
Note that you are not restricted to using serialised R objects as values; you can use them as keys; this is perfectly valid:
r$SET(redux::object_to_bin(1:10), "mydata")## [Redis: OK]
r$GET(redux::object_to_bin(1:10))## [1] "mydata"Beyond GET / SET / DEL, Redis
offers potentially better ways of holding things like lists using its
native data types. For example;
r$RPUSH("mylist2", 1:10)## [1] 10(the returned value 10 indicates that the list “mylist2”
is 10 elements long). There are lots of commands for operating on lists.
For example, you can do things like;
Redis documentation
rather than R’s semantics)
r$LINDEX("mylist2", 1)## [1] "2"
r$LSET("mylist2", 1, "carrot")## [Redis: OK]
r$LRANGE("mylist2", 0, -1)## [[1]]
## [1] "1"
##
## [[2]]
## [1] "carrot"
##
## [[3]]
## [1] "3"
##
## [[4]]
## [1] "4"
##
## [[5]]
## [1] "5"
##
## [[6]]
## [1] "6"
##
## [[7]]
## [1] "7"
##
## [[8]]
## [1] "8"
##
## [[9]]
## [1] "9"
##
## [[10]]
## [1] "10"
r$LRANGE("mylist2", 0, 2)## [[1]]
## [1] "1"
##
## [[2]]
## [1] "carrot"
##
## [[3]]
## [1] "3"
r$LLEN("mylist2")## [1] 10
r$LPOP("mylist2")## [1] "1"
r$RPOP("mylist2")## [1] "10"
r$LLEN("mylist2")## [1] 8Of course, each element of the list can be an R object if you run it
through object_to_string:
r$LPUSH("mylist2", redux::object_to_string(1:10))## [1] 9but you’ll be responsible for converting that back (and detecting / knowing that this needs doing)
dat <- r$LRANGE("mylist2", 0, 2)
dat## [[1]]
## [1] "A\n3\n262914\n197888\n5\nUTF-8\n238\n2\n1\n262153\n14\ncompact_intseq\n2\n1\n262153\n4\nbase\n2\n13\n1\n13\n254\n14\n3\n0x1.4p+3\n0x1p+0\n0x1p+0\n254\n"
##
## [[2]]
## [1] "carrot"
##
## [[3]]
## [1] "3"
dat[[1]] <- redux::string_to_object(dat[[1]])
dat## [[1]]
## [1] 1 2 3 4 5 6 7 8 9 10
##
## [[2]]
## [1] "carrot"
##
## [[3]]
## [1] "3"As with all functions in the redis_api object, all
functions and their arguments are described in the Redis
documentation.
Every command set to Redis costs a round trip; even over the loopback interface this can be expensive if done a very large number of times. Redis offers two ways of minimising this problem; pipelining and lua scripting. redux supports both.
To pipeline, use the pipeline method of the
hiredis object:
redis <- redux::redis
r$pipeline(
redis$PING(),
redis$PING())## [[1]]
## [Redis: PONG]
##
## [[2]]
## [Redis: PONG]Here, redis is a special object within the package that
implements all the Redis commands but only formats them for use rather
than sends them. The pipeline method collects these all up
and sends them to the server in a single batch, with the result returned
as a list.
If arguments are named, then the return value is named:
r$pipeline(
a = redis$INCR("x"),
b = redis$INCR("x"),
c = redis$DEL("x"))## $a
## [1] 1
##
## $b
## [1] 2
##
## $c
## [1] 1here a variable “x” was incremented twice and then deleted.
If you use pipelining you should read the Redis page on it
(http://redis.io/topics/pipelining) because there are a few
restrictions and cautions.
Generating very large numbers (or variable numbers) of commands with
the above interface will be difficult because pipeline uses
the dots argument. Instead, you can pass a list of commands to the
.commands argument of pipeline:
## [[1]]
## [Redis: PONG]
##
## [[2]]
## [Redis: PONG]
##
## [[3]]
## [Redis: PONG]
##
## [[4]]
## [Redis: PONG]On top of the key/value store aspect of Redis, it also offers a
publisher/subscriber model. Publishing with redux is
straightforward; use the PUBLISH method:
r$PUBLISH("mychannel", "hello")## [1] 0The return value here is the number of subscribers to that channel; in our case zero!
The SUBSCRIBE method should not be used as the client
cannot deal with messages directly (it is disabled in the interface to
prevent this).
Instead, use the subscribe (lower case) method. This
takes arguments:
channel: name or pattern of the channel/s to
subscribe to (scalar or vector).
transform: A function that takes each message and
processes it. Messages are R lists with elements: type,
pattern (if a pattern was used), channel and
value (see the Redis docs). Your transform function can
turn this into anything it wants, and may have side effects such as
printing to the screen, writing to a file, etc.
terminate: A termination criterion. given a
transformed message (i.e., the result of
transform(x)) return TRUE if we’re processing
messages. Optional, but if not used set n to a finite
number.
collect: logical indicating if transformed messages should be collected and returned on exit.
n: maximum number of messages to collect; once n
messages have been collected we will terminate regardless of
terminate.
pattern: logical indicating if channel should be
interpreted as a pattern.
envir: environment in which to evaluate transform
and terminate.
That all sounds a lot more complicated it really is. To collect all
messages on the "mychannel" channel, stopping after 100
messages or a message reading exactly “goodbye” you would write:
res <- r$subscribe("mychannel",
transform = function(x) x$value,
terminate = function(x) identical(x, "goodbye"),
n = 100)NOTE: you need to be careful here - hiredis
internally uses a blocking read which cannot be interrupted with Ctrl-C
once started unless a message is received on the channels being listened
to!
To test this out, we need a second process that will publish to the channel (or we’ll wait forever). This function will publish the first 20 values out of the Nile data set.
r <- redux::hiredis()
for (i in Nile[1:20]) {
Sys.sleep(.05)
r$PUBLISH("mychannel", i)
}
r$PUBLISH("mychannel", "goodbye")This file is at path_to_publisher (in R’s temporary
directory) and can be run with:
system2(file.path(R.home("bin"), "Rscript"), path_to_publisher,
wait = FALSE, stdout = FALSE, stderr = FALSE)to start the publisher.
Let’s add a little debugging information to the transform function, and set the subscriber off:
transform <- function(x) {
message(format(Sys.time(), "%Y-%m-%d %H:%M:%OS3"),
": got message: ",
x$value)
x$value
}
res <- r$subscribe("mychannel",
transform = transform,
terminate = function(x) identical(x, "goodbye"),
n = 100)## 2024-01-10 09:22:16.995: got message: 1120## 2024-01-10 09:22:17.188: got message: 1160## 2024-01-10 09:22:17.294: got message: 963## 2024-01-10 09:22:17.384: got message: 1210## 2024-01-10 09:22:17.436: got message: 1160## 2024-01-10 09:22:17.589: got message: 1160## 2024-01-10 09:22:17.763: got message: 813## 2024-01-10 09:22:17.856: got message: 1230## 2024-01-10 09:22:17.908: got message: 1370## 2024-01-10 09:22:17.997: got message: 1140## 2024-01-10 09:22:18.183: got message: 995## 2024-01-10 09:22:18.375: got message: 935## 2024-01-10 09:22:18.526: got message: 1110## 2024-01-10 09:22:18.694: got message: 994## 2024-01-10 09:22:18.830: got message: 1020## 2024-01-10 09:22:18.908: got message: 960## 2024-01-10 09:22:18.964: got message: 1180## 2024-01-10 09:22:19.026: got message: 799## 2024-01-10 09:22:19.077: got message: 958## 2024-01-10 09:22:19.161: got message: 1140## 2024-01-10 09:22:19.162: got message: goodbyeThe timestamps in the printed output show when the message was received (with fractional seconds so that this is more obvious since this only takes ~1s to complete).
The res object contains all the values, including the
“goodbye” that was our end-of-stream message:
unlist(res)## [1] "1120" "1160" "963" "1210" "1160" "1160" "813"
## [8] "1230" "1370" "1140" "995" "935" "1110" "994"
## [15] "1020" "960" "1180" "799" "958" "1140" "goodbye"Because redux exposes all of Redis, you can roll your
own data structures.
First, a generator object that sets up a new list at key
within the database r.
rlist <- function(..., key = "rlist", r = redux::hiredis()) {
dat <- vapply(c(...), redux::object_to_string, character(1))
r$RPUSH(key, dat)
ret <- list(r = r, key = key)
class(ret) <- "rlist"
ret
}Then some S3 methods that work with this object. I’ve only
implemented length and [[, but [
would be useful here too as would print.
length.rlist <- function(x) {
x$r$LLEN(x$key)
}
`[[.rlist` <- function(x, i, ...) {
redux::string_to_object(x$r$LINDEX(x$key, i - 1L))
}
`[[<-.rlist` <- function(x, i, value, ...) {
x$r$LSET(x$key, i - 1L, redux::object_to_string(value))
x
}Then we have this weird object we can add things to.
obj <- rlist(1:10)
length(obj) # 10## [1] 10
obj[[3]]## [1] 3
obj[[3]] <- "an element"
obj[[3]]## [1] "an element"The object has reference semantics so that assignment does not make a copy:
obj2 <- obj
obj2[[2]] <- obj2[[2]] * 2
obj[[2]] == obj2[[2]]## [1] TRUEFor a better version of this, see storr which does similar things to implement “indexable serialisation”
Redis allows storing and evaluating Lua scripts on the redis server. At this point it’s all getting a bit meta (using R to tell Redis to call another dynamic language that drives Redis) but this can be very useful - especially in avoiding race conditions (because a script is atomic) and avoiding roundtrips.
Describing how to write Lua scripts is out of scope for this document but is a bit fiddly. Here is a trivial one that returns the value of a key:
r$SET("key", "a")## [Redis: OK]
res <- r$EVAL("return redis.call('get', 'key')", 1L, "key", NULL)This can also be run by pushing the script into Redis and referring to it by SHA:
sha <- r$SCRIPT_LOAD("return redis.call('get', 'key')")
r$SCRIPT_EXISTS(sha)## [[1]]
## [1] 1and calling it like so:
r$EVALSHA(sha, 1, "key", NULL)## [1] "a"A more interesting example, setting, incrementing and getting a key (this is all do-able with redis commands)
lua <- '
local keyname = KEYS[1]
local value = ARGV[1]
redis.call("SET", keyname, value)
redis.call("INCR", keyname)
return redis.call("GET", keyname)'With the redis_scripts wrapper you can give friendly
names to a script:
obj <- redux::redis_scripts(r, set_and_incr = lua)And then call them by name:
res <- obj("set_and_incr", "foo", "10")
res## [1] "11"