SignalR Scaleout with Redis

In this tutorial, you will use Redis to distribute messages across a SignalR application that is deployed on two separate IIS instances.

Redis is an in-memory key-value store. It also supports a messaging system with a publish/subscribe model. The SignalR Redis backplane uses the pub/sub feature to forward messages to other servers.

For this tutorial, you will use three servers:

  • Two servers running Windows, which you will use to deploy a SignalR application.
  • One server running Linux, which you will use to run Redis. For the screenshots in this tutorial, I used Ubuntu 12.04 TLS.

If you don’t have three physical servers to use, you can create VMs on Hyper-V. Another option is to create VMs on Windows Azure.

Although this tutorial uses the official Redis implementation, there is also a Windows port of Redis from MSOpenTech. Setup and configuration are different, but otherwise the steps are the same.

Overview

Before we get to the detailed tutorial, here is a quick overview of what you will do.

  1. Install Redis and start the Redis server.
  2. Add these NuGet packages to your application:
  3. Create a SignalR application.
  4. Add the following code to Startup.cs to configure the backplane:
publicclassStartup{publicvoidConfiguration(IAppBuilder app){// Any connection or hub wire up and configuration should go hereGlobalHost.DependencyResolver.UseRedis("server", port,"password","AppName");
            app.MapSignalR();}}

Ubuntu on Hyper-V

Using Windows Hyper-V, you can easily create an Ubuntu VM on Windows Server.

Download the Ubuntu ISO from http://www.ubuntu.com.

In Hyper-V, add a new VM. In the Connect Virtual Hard Disk step, select Create a virtual hard disk.

In the Installation Options step, select Image file (.iso), click Browse, and browse to the Ubuntu installation ISO.

Install Redis

Follow the steps at http://redis.io/download to download and build Redis.

wget http://redis.googlecode.com/files/redis-2.6.12.tar.gz
tar xzf redis-2.6.12.tar.gz
cd redis-2.6.12
make

This builds the Redis binaries in the src directory.

By default, Redis does not require a password. To set a password, edit the redis.conf file, which is located in the root directory of the source code. (Make a backup copy of the file before you edit it!) Add the following directive to redis.conf:

requirepass YourStrongPassword1234

Now start the Redis server:

src/redis-server redis.conf

Open port 6379, which is the default port that Redis listens on. (You can change the port number in the configuration file.)

Create the SignalR Application

Create a SignalR application by following either of these tutorials:

Next, we’ll modify the chat application to support scaleout with Redis. First, add the SignalR.Redis NuGet package to your project. In Visual Studio, from the Tools menu, select Library Package Manager, then select Package Manager Console. In the Package Manager Console window, enter the following command:

Install-PackageMicrosoft.AspNet.SignalR.Redis

Next, open the Startup.cs file. Add the following code to the Configuration method:

publicclassStartup{publicvoidConfiguration(IAppBuilder app){// Any connection or hub wire up and configuration should go hereGlobalHost.DependencyResolver.UseRedis("server", port,"password","AppName");
            app.MapSignalR();}}
  • “server” is the name of the server that is running Redis.
  • port is the port number
  • “password” is the password that you defined in the redis.conf file.
  • “AppName” is any string. SignalR creates a Redis pub/sub channel with this name.

For example:

GlobalHost.DependencyResolver.UseRedis("redis-server.cloudapp.net",6379,"MyStrongPassword1234","ChatApp");

Deploy and Run the Application

Prepare your Windows Server instances to deploy the SignalR application.

Add the IIS role. Include “Application Development” features, including the WebSocket Protocol.

Also include the Management Service (listed under “Management Tools”).

Install Web Deploy 3.0. When you run IIS Manager, it will prompt you to install Microsoft Web Platform, or you can download the intstaller. In the Platform Installer, search for Web Deploy and install Web Deploy 3.0

Check that the Web Management Service is running. If not, start the service. (If you don’t see Web Management Service in the list of Windows services, make sure that you installed the Management Service when you added the IIS role.)

By default, the Web Management Service listens on TCP port 8172. In Windows Firewall, create a new inbound rule to allow TCP traffic on port 8172. For more information, see Configuring Firewall Rules. (If you are hosting the VMs on Windows Azure, you can do this directly in the Windows Azure portal. See How to Set Up Communication with a Virtual Machine.)

Now you are ready to deploy the Visual Studio project from your development machine to the server. In Solution Explorer, right-click the solution and click Publish.

For more detailed documentation about web deployment, see Web Deployment Content Map for Visual Studio and ASP.NET.

If you deploy the application to two servers, you can open each instance in a separate browser window and see that they each receive SignalR messages from the other. (Of course, in a production environment, the two servers would sit behind a load balancer.)

If you’re curious to see the messages that are sent to Redis, you can use the redis-cli client, which installs with Redis.

redis-cli -a password
SUBSCRIBE ChatApp

LINK: http://www.asp.net/signalr/overview/signalr-20/performance-and-scaling/scaleout-with-redis

Advertisements

Redis Benchmarks

How fast is Redis?

Redis includes the redis-benchmark utility that simulates running commands done by N clients at the same time sending M total queries (it is similar to the Apache’s ab utility). Below you’ll find the full output of a benchmark executed against a Linux box.

The following options are supported:

Usage: redis-benchmark [-h <host>] [-p <port>] [-c <clients>] [-n <requests]> [-k <boolean>]

 -h <hostname>      Server hostname (default 127.0.0.1)
 -p <port>          Server port (default 6379)
 -s <socket>        Server socket (overrides host and port)
 -c <clients>       Number of parallel connections (default 50)
 -n <requests>      Total number of requests (default 10000)
 -d <size>          Data size of SET/GET value in bytes (default 2)
 -k <boolean>       1=keep alive 0=reconnect (default 1)
 -r <keyspacelen>   Use random keys for SET/GET/INCR, random values for SADD
  Using this option the benchmark will get/set keys
  in the form mykey_rand:000000012456 instead of constant
  keys, the <keyspacelen> argument determines the max
  number of values for the random number. For instance
  if set to 10 only rand:000000000000 - rand:000000000009
  range will be allowed.
 -P <numreq>        Pipeline <numreq> requests. Default 1 (no pipeline).
 -q                 Quiet. Just show query/sec values
 --csv              Output in CSV format
 -l                 Loop. Run the tests forever
 -t <tests>         Only run the comma separated list of tests. The test
                    names are the same as the ones produced as output.
 -I                 Idle mode. Just open N idle connections and wait.

You need to have a running Redis instance before launching the benchmark. A typical example would be:

redis-benchmark -q -n 100000

Using this tool is quite easy, and you can also write your own benchmark, but as with any benchmarking activity, there are some pitfalls to avoid.

Running only a subset of the tests

You don’t need to run all the default tests every time you execute redis-benchmark. The simplest thing to select only a subset of tests is to use the -t option like in the following example:

$ redis-benchmark -t set,lpush -n 100000 -q
SET: 74239.05 requests per second
LPUSH: 79239.30 requests per second

In the above example we asked to just run test the SET and LPUSH commands, in quite mode (see the -q switch).

It is also possible to specify the command to benchmark directly like in the following example:

$ redis-benchmark -n 100000 -q script load "redis.call('set','foo','bar')"
script load redis.call('set','foo','bar'): 69881.20 requests per second

Selecting the size of the key space

By default the benchmark runs against a single key. In Redis the difference between such a synthetic benchmark and a real one is not huge since it is an in memory system, however it is possible to stress cache misses and in general to simulate a more real-world work load by using a large key space.

This is obtained by using the -r switch. For instance if I want to run one million of SET operations, using a random key for every operation out of 100k possible keys, I’ll use the following command line:

$ redis-cli flushall
OK

$ redis-benchmark -t set -r 100000 -n 1000000
====== SET ======
  1000000 requests completed in 13.86 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.76% `<=` 1 milliseconds
99.98% `<=` 2 milliseconds
100.00% `<=` 3 milliseconds
100.00% `<=` 3 milliseconds
72144.87 requests per second

$ redis-cli dbsize
(integer) 99993

Using pipelining

By default every client (the benchmark simulates 50 clients if not otherwise specified with -c) sends the next command only when the reply of the previous command is received, this means that the server will likely need a read call in order to read each command from every client. Also RTT is payed as well.

Redis supports /topics/pipelining, so it is possible to send multiple commands at once, a feature often exploited by real world applications. Redis pipelining is able to dramatically improve the number of operations per second a server is able do deliver.

This is an example of running the benchmark in a Macbook air 11″ using a pipeling of 16 commands:

$ redis-benchmark -n 1000000 -t set,get -P 16 -q
SET: 403063.28 requests per second
GET: 508388.41 requests per second

Using pipelining resulted into a sensible amount of more commands processed.

Pitfalls and misconceptions

The first point is obvious: the golden rule of a useful benchmark is to only compare apples and apples. Different versions of Redis can be compared on the same workload for instance. Or the same version of Redis, but with different options. If you plan to compare Redis to something else, then it is important to evaluate the functional and technical differences, and take them in account.

  • Redis is a server: all commands involve network or IPC roundtrips. It is meaningless to compare it to embedded data stores such as SQLite, Berkeley DB, Tokyo/Kyoto Cabinet, etc … because the cost of most operations is precisely dominated by network/protocol management.
  • Redis commands return an acknowledgment for all usual commands. Some other data stores do not (for instance MongoDB does not implicitly acknowledge write operations). Comparing Redis to stores involving one-way queries is only mildly useful.
  • Naively iterating on synchronous Redis commands does not benchmark Redis itself, but rather measure your network (or IPC) latency. To really test Redis, you need multiple connections (like redis-benchmark) and/or to use pipelining to aggregate several commands and/or multiple threads or processes.
  • Redis is an in-memory data store with some optional persistency options. If you plan to compare it to transactional servers (MySQL, PostgreSQL, etc …), then you should consider activating AOF and decide of a suitable fsync policy.
  • Redis is a single-threaded server. It is not designed to benefit from multiple CPU cores. People are supposed to launch several Redis instances to scale out on several cores if needed. It is not really fair to compare one single Redis instance to a multi-threaded data store.

A common misconception is that redis-benchmark is designed to make Redis performances look stellar, the throughput achieved by redis-benchmark being somewhat artificial, and not achievable by a real application. This is actually plain wrong.

The redis-benchmark program is a quick and useful way to get some figures and evaluate the performance of a Redis instance on a given hardware. However, by default, it does not represent the maximum throughput a Redis instance can sustain. Actually, by using pipelining and a fast client (hiredis), it is fairly easy to write a program generating more throughput than redis-benchmark. The default behavior of redis-benchmark is to achieve throughput by exploiting concurrency only (i.e. it creates several connections to the server). It does not use pipelining or any parallelism at all (one pending query per connection at most, and no multi-threading).

To run a benchmark using pipelining mode (and achieve higher throughputs), you need to explicitly use the -P option. Please note that it is still a realistic behavior since a lot of Redis based applications actively use pipelining to improve performance.

Finally, the benchmark should apply the same operations, and work in the same way with the multiple data stores you want to compare. It is absolutely pointless to compare the result of redis-benchmark to the result of another benchmark program and extrapolate.

For instance, Redis and memcached in single-threaded mode can be compared on GET/SET operations. Both are in-memory data stores, working mostly in the same way at the protocol level. Provided their respective benchmark application is aggregating queries in the same way (pipelining) and use a similar number of connections, the comparison is actually meaningful.

This perfect example is illustrated by the dialog between Redis (antirez) and memcached (dormando) developers.

antirez 1 – On Redis, Memcached, Speed, Benchmarks and The Toilet

dormando – Redis VS Memcached (slightly better bench)

antirez 2 – An update on the Memcached/Redis benchmark

You can see that in the end, the difference between the two solutions is not so staggering, once all technical aspects are considered. Please note both Redis and memcached have been optimized further after these benchmarks …

Finally, when very efficient servers are benchmarked (and stores like Redis or memcached definitely fall in this category), it may be difficult to saturate the server. Sometimes, the performance bottleneck is on client side, and not server-side. In that case, the client (i.e. the benchmark program itself) must be fixed, or perhaps scaled out, in order to reach the maximum throughput.

Factors impacting Redis performance

There are multiple factors having direct consequences on Redis performance. We mention them here, since they can alter the result of any benchmarks. Please note however, that a typical Redis instance running on a low end, non tuned, box usually provides good enough performance for most applications.

  • Network bandwidth and latency usually have a direct impact on the performance. It is a good practice to use the ping program to quickly check the latency between the client and server hosts is normal before launching the benchmark. Regarding the bandwidth, it is generally useful to estimate the throughput in Gbits/s and compare it to the theoretical bandwidth of the network. For instance a benchmark setting 4 KB strings in Redis at 100000 q/s, would actually consume 3.2 Gbits/s of bandwidth and probably fit with a 10 GBits/s link, but not a 1 Gbits/s one. In many real world scenarios, Redis throughput is limited by the network well before being limited by the CPU. To consolidate several high-throughput Redis instances on a single server, it worth considering putting a 10 Gbits/s NIC or multiple 1 Gbits/s NICs with TCP/IP bonding.
  • CPU is another very important factor. Being single-threaded, Redis favors fast CPUs with large caches and not many cores. At this game, Intel CPUs are currently the winners. It is not uncommon to get only half the performance on an AMD Opteron CPU compared to similar Nehalem EP/Westmere EP/Sandy bridge Intel CPUs with Redis. When client and server run on the same box, the CPU is the limiting factor with redis-benchmark.
  • Speed of RAM and memory bandwidth seem less critical for global performance especially for small objects. For large objects (>10 KB), it may become noticeable though. Usually, it is not really cost effective to buy expensive fast memory modules to optimize Redis.
  • Redis runs slower on a VM. Virtualization toll is quite high because for many common operations, Redis does not add much overhead on top of the required system calls and network interruptions. Prefer to run Redis on a physical box, especially if you favor deterministic latencies. On a state-of-the-art hypervisor (VMWare), result of redis-benchmark on a VM through the physical network is almost divided by 2 compared to the physical machine, with some significant CPU time spent in system and interruptions.
  • When the server and client benchmark programs run on the same box, both the TCP/IP loopback and unix domain sockets can be used. It depends on the platform, but unix domain sockets can achieve around 50% more throughput than the TCP/IP loopback (on Linux for instance). The default behavior of redis-benchmark is to use the TCP/IP loopback.
  • The performance benefit of unix domain sockets compared to TCP/IP loopback tends to decrease when pipelining is heavily used (i.e. long pipelines).
  • When an ethernet network is used to access Redis, aggregating commands using pipelining is especially efficient when the size of the data is kept under the ethernet packet size (about 1500 bytes). Actually, processing 10 bytes, 100 bytes, or 1000 bytes queries almost result in the same throughput. See the graph below.

Data size impact

  • On multi CPU sockets servers, Redis performance becomes dependant on the NUMA configuration and process location. The most visible effect is that redis-benchmark results seem non deterministic because client and server processes are distributed randomly on the cores. To get deterministic results, it is required to use process placement tools (on Linux: taskset or numactl). The most efficient combination is always to put the client and server on two different cores of the same CPU to benefit from the L3 cache. Here are some results of 4 KB SET benchmark for 3 server CPUs (AMD Istanbul, Intel Nehalem EX, and Intel Westmere) with different relative placements. Please note this benchmark is not meant to compare CPU models between themselves (CPUs exact model and frequency are therefore not disclosed).

NUMA chart

  • With high-end configurations, the number of client connections is also an important factor. Being based on epoll/kqueue, Redis event loop is quite scalable. Redis has already been benchmarked at more than 60000 connections, and was still able to sustain 50000 q/s in these conditions. As a rule of thumb, an instance with 30000 connections can only process half the throughput achievable with 100 connections. Here is an example showing the throughput of a Redis instance per number of connections:

connections chart

  • With high-end configurations, it is possible to achieve higher throughput by tuning the NIC(s) configuration and associated interruptions. Best throughput is achieved by setting an affinity between Rx/Tx NIC queues and CPU cores, and activating RPS (Receive Packet Steering) support. More information in this thread. Jumbo frames may also provide a performance boost when large objects are used.
  • Depending on the platform, Redis can be compiled against different memory allocators (libc malloc, jemalloc, tcmalloc), which may have different behaviors in term of raw speed, internal and external fragmentation. If you did not compile Redis by yourself, you can use the INFO command to check the mem_allocator field. Please note most benchmarks do not run long enough to generate significant external fragmentation (contrary to production Redis instances).

Other things to consider

One important goal of any benchmark is to get reproducible results, so they can be compared to the results of other tests.

  • A good practice is to try to run tests on isolated hardware as far as possible. If it is not possible, then the system must be monitored to check the benchmark is not impacted by some external activity.
  • Some configurations (desktops and laptops for sure, some servers as well) have a variable CPU core frequency mechanism. The policy controlling this mechanism can be set at the OS level. Some CPU models are more aggressive than others at adapting the frequency of the CPU cores to the workload. To get reproducible results, it is better to set the highest possible fixed frequency for all the CPU cores involved in the benchmark.
  • An important point is to size the system accordingly to the benchmark. The system must have enough RAM and must not swap. On Linux, do not forget to set the overcommit_memory parameter correctly. Please note 32 and 64 bits Redis instances have not the same memory footprint.
  • If you plan to use RDB or AOF for your benchmark, please check there is no other I/O activity in the system. Avoid putting RDB or AOF files on NAS or NFS shares, or on any other devices impacting your network bandwidth and/or latency (for instance, EBS on Amazon EC2).
  • Set Redis logging level (loglevel parameter) to warning or notice. Avoid putting the generated log file on a remote filesystem.
  • Avoid using monitoring tools which can alter the result of the benchmark. For instance using INFO at regular interval to gather statistics is probably fine, but MONITOR will impact the measured performance significantly.

Benchmark results on different virtualized and bare metal servers.

  • The test was done with 50 simultaneous clients performing 2 million requests.
  • Redis 2.6.14 is used for all the tests.
  • Test executed using the loopback interface.
  • Test executed using a key space of 1 million keys.
  • Test executed with and without pipelining (16 commands pipeline).

Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (with pipelining)

$ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -P 16 -q
SET: 552028.75 requests per second
GET: 707463.75 requests per second
LPUSH: 767459.75 requests per second
LPOP: 770119.38 requests per second

Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (without pipelining)

$ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -q
SET: 122556.53 requests per second
GET: 123601.76 requests per second
LPUSH: 136752.14 requests per second
LPOP: 132424.03 requests per second

Linode 2048 instance (with pipelining)

$ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -q -P 16
SET: 195503.42 requests per second
GET: 250187.64 requests per second
LPUSH: 230547.55 requests per second
LPOP: 250815.16 requests per second

Linode 2048 instance (without pipelining)

$ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -q
SET: 35001.75 requests per second
GET: 37481.26 requests per second
LPUSH: 36968.58 requests per second
LPOP: 35186.49 requests per second

More detailed tests without pipelining

$ redis-benchmark -n 100000

====== SET ======
  100007 requests completed in 0.88 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

58.50% <= 0 milliseconds
99.17% <= 1 milliseconds
99.58% <= 2 milliseconds
99.85% <= 3 milliseconds
99.90% <= 6 milliseconds
100.00% <= 9 milliseconds
114293.71 requests per second

====== GET ======
  100000 requests completed in 1.23 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

43.12% <= 0 milliseconds
96.82% <= 1 milliseconds
98.62% <= 2 milliseconds
100.00% <= 3 milliseconds
81234.77 requests per second

====== INCR ======
  100018 requests completed in 1.46 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

32.32% <= 0 milliseconds
96.67% <= 1 milliseconds
99.14% <= 2 milliseconds
99.83% <= 3 milliseconds
99.88% <= 4 milliseconds
99.89% <= 5 milliseconds
99.96% <= 9 milliseconds
100.00% <= 18 milliseconds
68458.59 requests per second

====== LPUSH ======
  100004 requests completed in 1.14 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

62.27% <= 0 milliseconds
99.74% <= 1 milliseconds
99.85% <= 2 milliseconds
99.86% <= 3 milliseconds
99.89% <= 5 milliseconds
99.93% <= 7 milliseconds
99.96% <= 9 milliseconds
100.00% <= 22 milliseconds
100.00% <= 208 milliseconds
88109.25 requests per second

====== LPOP ======
  100001 requests completed in 1.39 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

54.83% <= 0 milliseconds
97.34% <= 1 milliseconds
99.95% <= 2 milliseconds
99.96% <= 3 milliseconds
99.96% <= 4 milliseconds
100.00% <= 9 milliseconds
100.00% <= 208 milliseconds
71994.96 requests per second

Notes: changing the payload from 256 to 1024 or 4096 bytes does not change the numbers significantly (but reply packets are glued together up to 1024 bytes so GETs may be slower with big payloads). The same for the number of clients, from 50 to 256 clients I got the same numbers. With only 10 clients it starts to get a bit slower.

You can expect different results from different boxes. For example a low profile box like Intel core duo T5500 clocked at 1.66 GHz running Linux 2.6 will output the following:

$ ./redis-benchmark -q -n 100000
SET: 53684.38 requests per second
GET: 45497.73 requests per second
INCR: 39370.47 requests per second
LPUSH: 34803.41 requests per second
LPOP: 37367.20 requests per second

Another one using a 64 bit box, a Xeon L5420 clocked at 2.5 GHz:

$ ./redis-benchmark -q -n 100000
PING: 111731.84 requests per second
SET: 108114.59 requests per second
GET: 98717.67 requests per second
INCR: 95241.91 requests per second
LPUSH: 104712.05 requests per second
LPOP: 93722.59 requests per second

Example of benchmark results with optimized high-end server hardware

  • Redis version 2.4.2
  • Default number of connections, payload size = 256
  • The Linux box is running SLES10 SP3 2.6.16.60-0.54.5-smp, CPU is 2 x Intel X5670 @ 2.93 GHz.
  • Text executed while running redis server and benchmark client on the same CPU, but different cores.

Using a unix domain socket:

$ numactl -C 6 ./redis-benchmark -q -n 100000 -s /tmp/redis.sock -d 256
PING (inline): 200803.22 requests per second
PING: 200803.22 requests per second
MSET (10 keys): 78064.01 requests per second
SET: 198412.69 requests per second
GET: 198019.80 requests per second
INCR: 200400.80 requests per second
LPUSH: 200000.00 requests per second
LPOP: 198019.80 requests per second
SADD: 203665.98 requests per second
SPOP: 200803.22 requests per second
LPUSH (again, in order to bench LRANGE): 200000.00 requests per second
LRANGE (first 100 elements): 42123.00 requests per second
LRANGE (first 300 elements): 15015.02 requests per second
LRANGE (first 450 elements): 10159.50 requests per second
LRANGE (first 600 elements): 7548.31 requests per second

Using the TCP loopback:

$ numactl -C 6 ./redis-benchmark -q -n 100000 -d 256
PING (inline): 145137.88 requests per second
PING: 144717.80 requests per second
MSET (10 keys): 65487.89 requests per second
SET: 142653.36 requests per second
GET: 142450.14 requests per second
INCR: 143061.52 requests per second
LPUSH: 144092.22 requests per second
LPOP: 142247.52 requests per second
SADD: 144717.80 requests per second
SPOP: 143678.17 requests per second
LPUSH (again, in order to bench LRANGE): 143061.52 requests per second
LRANGE (first 100 elements): 29577.05 requests per second
LRANGE (first 300 elements): 10431.88 requests per second
LRANGE (first 450 elements): 7010.66 requests per second
LRANGE (first 600 elements): 5296.61 requests per second

LINK: http://redis.io/topics/benchmarks

Designing a NoSQL Database using Redis

This page illustrates a good solution on how to design a simple Blog application with Redis using advanced features of ServiceStack’s C# Redis Client to provide fast, simple and elegant solutions to real world scenarios.

All Redis Blog application pages

Designing a NoSQL Database using Redis

Oren Eini from the popular .NET blog http://ayende.com/Blog/, is putting together a series of blog posts explaining how to go about designing a simple blog application using a NoSQL database. Although he’s using his RavenDB as a reference implementation, this example applies equally well to Redis and other NoSQL variants. Some solutions will vary based on the advanced features of each NoSQL solution but the ‘core data models’ should remain the same.

I will try to add my own thoughts on and where possible show how you can use the advanced features in Redis the C# Client to provide simple, fast and effective solutions. The entire source code for this example is available in its simplest form at: BlogPostExample.cs. I will be re-factoring this solution in what I consider a ‘best practices approach’ for large applications where I will shove all Redis access behind a repository (so the client doesn’t know it’s even being used) which I will be maintaining at: BlogPostBestPractice.cs

Modeling Entities in a NoSQL Database

If you’ve spent a lot of time building solutions with an RDBMS back-end it can be hard to know which part of your schema is due to the problem domain and which part is the result of an implementation constraint trying to map your ideal domain model onto a relational tabular structure.

My approach before designing any system is to map out the ideal domain model we need to build in order before I reach for an IDE or a db gui schema creator. Unfortunately creating POCO types is just so damn quick in VS.NET/C#/R# that I’ve ditched the pencil and paper a long time ago and jump right into using C# automatic properties like a kind of light-weight DSL ripping out entities quicker than I can draw crows feet 🙂

If you are like me and prefer to design your domain models from POCO types rather than creating RDBMS tables than you’re in good shape in implementing a NoSQL solution as the time when you usually morph your pristine domain models into a tabular structure peppering it with Primary and Foreign keys can now saved and put towards a longer lunch break as most of the times you can skip this step entirely.

The schema-less nature of NoSQL databases means you can pretty much store your domain models as-is. You will still need to identify your distinct entities from your Key Value Objects. A good guide I use to help with this is whether the Model only makes sense in the context its parent and whether it is ‘co-owned’ or referenced by another entity. This is where we start pulling the domain model apart, basically you just replace the collection of strongly-typed entities to a collection of entity ids. Effectively you can think of this like foreign-keys but at a much higher level as you only pull it apart of the domain model when you want to manage the entities independently of each other, not as dictated by your schema.

Taking the User model in it’s most simplest form. A User has many blogs, now as you may want to view a list of blogs outside the context of a User (e.g. view a list of newly created blogs, most popular blogs for a category, etc). It becomes a good candidate to being promoted a first class entity.

//BeforepublicclassUser{
  publicintId{get;set;}
  publicstringName{get;set;}
  publicList<Blog>Blogs{get;set;}}//AfterpublicclassUser{
  publicintId{get;set;}
  publicstringName{get;set;}
  publicList<int>BlogIds{get;set;}}

With only a running redis-server instance running and the C# client, the full source to persist and retrieve a list of users is only:

var redis =newRedisClient();using(var redisUsers = redisClient.GetTypedClient<User>()){
        redisUsers.Store(newUser{Id= redisUsers.GetNextSequence(),Name="ayende"});
        redisUsers.Store(newUser{Id= redisUsers.GetNextSequence(),Name="mythz"});

        var allUsers = redisUsers.GetAll();

        //Recursively print the values of the POCO (To use T.Dump() Extension method yourself see: http://www.servicestack.net/mythz_blog/?p=202)
        Console.WriteLine(allUsers.Dump());}/*Output
[
        {
                Id: 1,
                Name: ayende,
                BlogIds: []
        },
        {
                Id: 2,
                Name: mythz,
                BlogIds: []
        }
]
 */

Ayende has outlined a few scenarios that the Blog application should support.

  • Main page: show list of blogs
  • Main page: show list of recent posts
  • Main page: show list of recent comments
  • Main page: show tag cloud for posts
  • Main page: show categories
  • Post page: show post and all comments
  • Post page: add comment to post
  • Tag page: show all posts for tag
  • Categories page: show all posts for category

The full source code for the Redis solution for each of these scenarios is available here. Although I will go through each solution in a little more detail below.

Main page: show list of blogs

Before we can show a list of blogs, we need to add some first. Here I make effective use of the Redis client’s unique sequence that is maintained for each type.

In identifying my entities I have a general preference for ‘automated ids’ which are either sequential integer ids or Guids – if the entities are going to be distributed across multiple data stores.

Apart from that persisting an object is just a straight forward process of serializing the object graph into a text-serialization format. By default, the Redis Client uses TypeSerializer’s JSV format as it’s the fastest and most compact text-format for .NET POCO types.

//Retrieve strongly-typed Redis clients that let's you natively persist POCO'susing(var redisUsers = redisClient.GetTypedClient<User>())using(var redisBlogs = redisClient.GetTypedClient<Blog>()){
        //Create the user, getting a unique User Id from the User sequence.
        var mythz =newUser{Id= redisUsers.GetNextSequence(),Name="Demis Bellot"};

        //create some blogs using unique Ids from the Blog sequence. Also adding references
        var mythzBlogs =newList<Blog>
        {
                newBlog
                {
                        Id= redisBlogs.GetNextSequence(),
                        UserId= mythz.Id,
                        UserName= mythz.Name,
                        Tags=newList<string>{"Architecture",".NET","Redis"},
                },
                newBlog
                {
                        Id= redisBlogs.GetNextSequence(),
                        UserId= mythz.Id,
                        UserName= mythz.Name,
                        Tags=newList<string>{"Music","Twitter","Life"},
                },
        };
        //Add the blog references
        mythzBlogs.ForEach(x => mythz.BlogIds.Add(x.Id));

        //Store the user and their blogs
        redisUsers.Store(mythz);
        redisBlogs.StoreAll(mythzBlogs);

        //retrieve all blogs
        var blogs = redisBlogs.GetAll();

        Console.WriteLine(blogs.Dump());}/*Output
[
        {
                Id: 1,
                UserId: 1,
                UserName: Demis Bellot,
                Tags: 
                [
                        Architecture,
                        .NET,
                        Redis
                ],
                BlogPostIds: []
        },
        {
                Id: 2,
                UserId: 1,
                UserName: Demis Bellot,
                Tags: 
                [
                        Music,
                        Twitter,
                        Life
                ],
                BlogPostIds: []
        }
]
*/

Main page: show list of recent posts

Main page: show list of recent comments

For this scenario we can take advantage of Redis’s LTRIM’ing operation to maintain custom rolling lists. The richness of Redis list operations also allow us to prepend or append at either end of the list which we take advantage of in this example.

//Get strongly-typed clientsusing(var redisBlogPosts = redisClient.GetTypedClient<BlogPost>())using(var redisComments = redisClient.GetTypedClient<BlogPostComment>()){
        //To keep this example let's pretend this is a new list of blog posts
        var newIncomingBlogPosts = redisBlogPosts.GetAll();

        //Let's get back an IList<BlogPost> wrapper around a Redis server-side List.
        var recentPosts = redisBlogPosts.Lists["urn:BlogPost:RecentPosts"];
        var recentComments = redisComments.Lists["urn:BlogPostComment:RecentComments"];

        foreach(var newBlogPost in newIncomingBlogPosts)
        {
                //Prepend the new blog posts to the start of the 'RecentPosts' list
                recentPosts.Prepend(newBlogPost);

                //Prepend all the new blog post comments to the start of the 'RecentComments' list
                newBlogPost.Comments.ForEach(recentComments.Prepend);
        }

        //Make this a Rolling list by only keep the latest 3 posts and comments
        recentPosts.Trim(0,2);
        recentComments.Trim(0,2);

        //Print out the last 3 posts:
        Console.WriteLine(recentPosts.GetAll().Dump());
        /* Output: 
        [
                {
                        Id: 2,
                        BlogId: 2,
                        Title: Redis,
                        Categories: 
                        [
                                NoSQL,
                                Cache
                        ],
                        Tags: 
                        [
                                Redis,
                                NoSQL,
                                Scalability,
                                Performance
                        ],
                        Comments: 
                        [
                                {
                                        Content: First Comment!,
                                        CreatedDate: 2010-04-20T22:14:02.755878Z
                                }
                        ]
                },
                {
                        Id: 1,
                        BlogId: 1,
                        Title: RavenDB,
                        Categories: 
                        [
                                NoSQL,
                                DocumentDB
                        ],
                        Tags: 
                        [
                                Raven,
                                NoSQL,
                                JSON,
                                .NET
                        ],
                        Comments: 
                        [
                                {
                                        Content: First Comment!,
                                        CreatedDate: 2010-04-20T22:14:02.755878Z
                                },
                                {
                                        Content: Second Comment!,
                                        CreatedDate: 2010-04-20T22:14:02.755878Z
                                }
                        ]
                },
                {
                        Id: 4,
                        BlogId: 2,
                        Title: Couch Db,
                        Categories: 
                        [
                                NoSQL,
                                DocumentDB
                        ],
                        Tags: 
                        [
                                CouchDb,
                                NoSQL,
                                JSON
                        ],
                        Comments: 
                        [
                                {
                                        Content: First Comment!,
                                        CreatedDate: 2010-04-20T22:14:02.755878Z
                                }
                        ]
                }
        ]
        */

        Console.WriteLine(recentComments.GetAll().Dump());
        /* Output:
        [
                {
                        Content: First Comment!,
                        CreatedDate: 2010-04-20T20:32:42.2970956Z
                },
                {
                        Content: First Comment!,
                        CreatedDate: 2010-04-20T20:32:42.2970956Z
                },
                {
                        Content: First Comment!,
                        CreatedDate: 2010-04-20T20:32:42.2970956Z
                }
        ]
         */}

Main page: show tag cloud for posts

Redis Sorted Sets provide the perfect data structure to maintain a Tag cloud of all tags. It’s very fast, elegant structure which provides custom-specific operations to maintain and sort the data.

//Get strongly-typed clientsusing(var redisBlogPosts = redisClient.GetTypedClient<BlogPost>()){
        var newIncomingBlogPosts = redisBlogPosts.GetAll();

        foreach(var newBlogPost in newIncomingBlogPosts)
        {
                //For every tag in each new blog post, increment the number of times each Tag has occurred 
                newBlogPost.Tags.ForEach(x =>
                        redisClient.IncrementItemInSortedSet("urn:TagCloud", x,1));
        }

        //Show top 5 most popular tags with their scores
        var tagCloud = redisClient.GetRangeWithScoresFromSortedSetDesc("urn:TagCloud",0,4);
        Console.WriteLine(tagCloud.Dump());}/* Output:
[
        [
                NoSQL,
                 4
        ],
        [
                Scalability,
                 2
        ],
        [
                JSON,
                 2
        ],
        [
                Redis,
                 1
        ],
        [
                Raven,
                 1
        ],
]
*/

Main page: show categories

To keep a unique list of categories the right structure to use is a Set. This allows you to freely add a value multiple times and there will never be any duplicates as only one of each value is stored.

using(var redisBlogPosts = redisClient.GetTypedClient<BlogPost>()){
        var blogPosts = redisBlogPosts.GetAll();

        foreach(var blogPost in blogPosts)
        {
                blogPost.Categories.ForEach(x =>
                          redisClient.AddToSet("urn:Categories", x));
        }

        var uniqueCategories = redisClient.GetAllFromSet("urn:Categories");
        Console.WriteLine(uniqueCategories.Dump());
        /* Output:
        [
                DocumentDB,
                NoSQL,
                Cluster,
                Cache
        ]
         */}

Post page: show post and all comments

There is nothing special to do here since comments are Key Value Objects they are stored and retrieved with the post, so retrieving the post retrieves it’s comments as well.

var postId =1;using(var redisBlogPosts = redisClient.GetTypedClient<BlogPost>()){
        var selectedBlogPost = redisBlogPosts.GetById(postId.ToString());

        Console.WriteLine(selectedBlogPost.Dump());
        /* Output:
        {
                Id: 1,
                BlogId: 1,
                Title: RavenDB,
                Categories: 
                [
                        NoSQL,
                        DocumentDB
                ],
                Tags: 
                [
                        Raven,
                        NoSQL,
                        JSON,
                        .NET
                ],
                Comments: 
                [
                        {
                                Content: First Comment!,
                                CreatedDate: 2010-04-20T21:26:31.9918236Z
                        },
                        {
                                Content: Second Comment!,
                                CreatedDate: 2010-04-20T21:26:31.9918236Z
                        }
                ]
        }
        */}

Post page: add comment to post

Modifying an entity are one of the strengths of a schema-less data store. Adding a comment is as simple as

  • retrieving it’s parent post
  • modifying the POCO entity in memory by adding a comment to the existing list
  • then saving the entity.
var postId =1;using(var redisBlogPosts = redisClient.GetTypedClient<BlogPost>()){
        var blogPost = redisBlogPosts.GetById(postId.ToString());
        blogPost.Comments.Add(
                newBlogPostComment{Content="Third Post!",CreatedDate=DateTime.UtcNow});
        redisBlogPosts.Store(blogPost);

        var refreshBlogPost = redisBlogPosts.GetById(postId.ToString());
        Console.WriteLine(refreshBlogPost.Dump());
        /* Output:
        {
                Id: 1,
                BlogId: 1,
                Title: RavenDB,
                Categories: 
                [
                        NoSQL,
                        DocumentDB
                ],
                Tags: 
                [
                        Raven,
                        NoSQL,
                        JSON,
                        .NET
                ],
                Comments: 
                [
                        {
                                Content: First Comment!,
                                CreatedDate: 2010-04-20T21:32:39.9688707Z
                        },
                        {
                                Content: Second Comment!,
                                CreatedDate: 2010-04-20T21:32:39.9688707Z
                        },
                        {
                                Content: Third Post!,
                                CreatedDate: 2010-04-20T21:32:40.2688879Z
                        }
                ]
        }
        */}

Tag page: show all posts for tag

Basically in order to view all the posts for a particular category we’ll need to provide a reverse-index by adding all matching post ids into a ‘Category > Post Id’ Set.

From there it’s just a matter of performing a batch request fetching all the Posts with the supplied Ids:

using(var redisBlogPosts = redisClient.GetTypedClient<BlogPost>()){
        var newIncomingBlogPosts = redisBlogPosts.GetAll();

        foreach(var newBlogPost in newIncomingBlogPosts)
        {
                //For each post add it's Id into each of it's 'Cateogry > Posts' index
                newBlogPost.Categories.ForEach(x =>
                          redisClient.AddToSet("urn:Category:"+ x, newBlogPost.Id.ToString()));
        }

        //Retrieve all the post ids for the category you want to view
        var documentDbPostIds = redisClient.GetAllFromSet("urn:Category:DocumentDB");

        //Make a batch call to retrieve all the posts containing the matching ids 
        //(i.e. the DocumentDB Category posts)
        var documentDbPosts = redisBlogPosts.GetByIds(documentDbPostIds);

        Console.WriteLine(documentDbPosts.Dump());
        /* Output:
        [
                {
                        Id: 4,
                        BlogId: 2,
                        Title: Couch Db,
                        Categories: 
                        [
                                NoSQL,
                                DocumentDB
                        ],
                        Tags: 
                        [
                                CouchDb,
                                NoSQL,
                                JSON
                        ],
                        Comments: 
                        [
                                {
                                        Content: First Comment!,
                                        CreatedDate: 2010-04-20T21:38:24.6305842Z
                                }
                        ]
                },
                {
                        Id: 1,
                        BlogId: 1,
                        Title: RavenDB,
                        Categories: 
                        [
                                NoSQL,
                                DocumentDB
                        ],
                        Tags: 
                        [
                                Raven,
                                NoSQL,
                                JSON,
                                .NET
                        ],
                        Comments: 
                        [
                                {
                                        Content: First Comment!,
                                        CreatedDate: 2010-04-20T21:38:24.6295842Z
                                },
                                {
                                        Content: Second Comment!,
                                        CreatedDate: 2010-04-20T21:38:24.6295842Z
                                }
                        ]
                }
        ]
         */}

LINK GỐC: https://code.google.com/p/servicestack/wiki/DesigningNoSqlDatabase

Tìm Hiểu Redis – Phần 1

1. Giới thiệu

Redis là hệ thống lưu trữ key-value với rất nhiều tính năng và được sử dụng rộng rãi. Redis nổi bật bởi việc hỗ trợ nhiều cấu trúc dữ liệu cơ bản (hash, list, set, sorted set, string), đồng thời cho phép scripting bằng ngôn ngữ lua. Bên cạnh lưu trữ key-value trên RAM với hiệu năng cao, redis còn hỗ trợ lưu trữ dữ liệu trên đĩa cứng (persistent redis) cho phép phục hồi dữ liệu khi gặp sự cố. Ngoài tính năng replicatation (sao chép giữa master-client), tính năng cluster (sao lưu master-master) cũng đang được phát triển . Để sử dụng một cách hiệu quả những tính năng redis hỗ trợ cũng như vận hành redis với hiệu suất cao nhất thì việc am hiểu hệ thống lưu trữ này là điều không thể thiếu. Chính vì lý do này, mình quyết định tìm hiểu mã nguồn redis. Loạt bài viết về redis này tóm tắt những điều mình tìm hiểu được từ việc đọc mã nguồn của redis.

2. Khái quát

Bạn có thể clone mã nguồn redis về máy tính mình bằng câu lệnh dưới đây:

git clone https://github.com/antirez/redis.git

Trước hết là một số thống kê nho nhỏ về redis (tại thời điểm bài viết): * Số lượng file mã nguồn: 55

ls *.c | wc -l
55
  • Số lượng file header: 30
    ls *.h | wc -l
    30
  • Tổng số dòng code: 43829
    wc -l *.[ch]
    341    adlist.c     197   pqsort.c            228   sha1.c          810   dict.c
    93     adlist.h     40    pqsort.h            17    sha1.h          173   dict.h
    435    ae.c         359   pubsub.c            169   slowlog.c       124   endianconv.c
    130    ae_epoll.c   93    rand.c              47    slowlog.h       64    endianconv.h
    315    ae_evport.c  38    rand.h              50    solarisfixes.h  52    fmacros.h
    118    ae.h         1230  rdb.c               530   sort.c          759   help.h
    132    ae_kqueue.c  114   rdb.h               144   syncio.c        483   intset.c
    99     ae_select.c  683   redis-benchmark.c   57    testhelp.h      50    intset.h
    441    anet.c       3008  redis.c             761   t_hash.c        295   lzf_c.c
    60     anet.h       218   redis-check-aof.c   1149  t_list.c        150   lzf_d.c
    1178   aof.c        768   redis-check-dump.c  913   t_set.c         100   lzf.h
    47     asciilogo.h  1556  redis-cli.c         459   t_string.c      159   lzfP.h
    220    bio.c        1517  redis.h             2205  t_zset.c        279   memtest.c
    41     bio.h        52    release.c           520   util.c          323   multi.c
    412    bitops.c     3     release.h           41    util.h          1444  networking.c
    2866   cluster.c    1658  replication.c       1     version.h       128   notify.c
    1726   config.c     198   rio.c               1534  ziplist.c       580   object.c
    195    config.h     104   rio.h               46    ziplist.h
    88     crc16.c      1065  scripting.c         467   zipmap.c
    191    crc64.c      732   sds.c               49    zipmap.h
    8      crc64.h      99    sds.h               351   zmalloc.c
    815    db.c         3160  sentinel.c          85    zmalloc.h
    929    debug.c      261   setproctitle.c
    43829  total

Một số thư viện được sử dụng: jemalloclinenoiselua

3. Các modules

Redis bao gồm các module sau:

  • Framework hỗ trợ xử lý bất đồng bộ, networking: ae, anet
  • Mô tả dữ liệu: sds.c, t_hash.c, t_list.c, t_string.c, t_zset.c, object.c, notify.c (pub-sub)
  • Lưu trữ dữ liệu, cơ sở dữ liệu: db.c, dict.c, ziplist.c, zipmap.c, adlist.c
  • Module hỗ trợ IO/persistent redis: rdb.c, aof.c, bio.c, rio.c
  • Utilities: crc16.c, crc64.c, pqsort.c, lzf_c.c, lzf_d.c

Mình sẽ lần lượt giới thiệu các modules trong các bài viết sau. Ở bài viết này, mình sẽ tập trung vào module IO/persistent redis.

4. Persistent redis

Bên cạnh việc lưu key-value trên bộ nhớ RAM, Redis có 2 background threads chuyên làm nhiệm vụ định kỳ ghi dữ liệu lên đĩa cứng.

Có 2 loại file được ghi xuống đĩa cứng:

  • RDB
  • AOF

RDB lưu dữ liệu dưới dạng đã mã hóa. AOF lưu lại toàn bộ dưới liệu dưới dạng command, giống như command mà redis client gửi đến server để thao tác bằng cách ghi đè xuống cuối file.

File rdb có thể coi là một snapshot của cơ sở dữ liệu tại một thời điểm nhất định. File dữ liệu này được dùng với 2 mục đích

  • Cho phép redis có thể phục hồi lại dữ liệu trên memory bằng việc đọc file
  • Bản thân dữ liệu được ghi ra file rdb sẽ được gửi đến các redis slave server, phục vụ mục đích sao lưu server.

Dữ liệu ghi ra file rdb được chỉnh sửa và mã hóa để giảm kích thước ghi trên đĩa, đồng thời tằng tốc độ replication. Cụ thể định dạng của file rdb như sau.

Với những key ngắn, việc dùng 32 bit để mô tả key là thừa thãi, do vậy redis quy định những key ngắn được mã hóa sử dụng 2 bit đầu tiên của 1 byte. Cụ thể: 00|000000 => 00, độ dài dữ liệu mô tả bởi 6 bit 01|000000 00000000 => 01, độ dài dữ liệu là 14 bit 10|000000 [số 32 bit] => 1 chuỗi độ dài 32 bit sẽ theo sau 11|000000 => obj được encode đặc biệt sẽ theo sau byte này. 6 bit sau sẽ xác định kiểu object.

Kiểu object ở đây cụ thể như sau:

rdb.c
1
2
3
4
5
6
7
/* When a length of a string object stored on disk has the first two bits
 * set, the remaining two bits specify a special encoding for the object
 * accordingly to the following defines: */
#define REDIS_RDB_ENC_INT8 0        /* 8 bit signed integer */
#define REDIS_RDB_ENC_INT16 1       /* 16 bit signed integer */
#define REDIS_RDB_ENC_INT32 2       /* 32 bit signed integer */
#define REDIS_RDB_ENC_LZF 3         /* string compressed with FASTLZ */

Với AOF file, các command sẽ được nhóm thành các block. Các block được tổ chức dưới dạng danh sách liên kết. Mỗi block có độ lớn 10MB là vì trong trường hợp redis server chịu tải cao, số lượng key được cập nhật lớn, nếu kích thước buffer lớn, việc realloc buffer dùng cho các command với tốc độ lớn không đảm bảo.

Trong trường hợp file rdb, redis fork 1 process con và thực hiện ghi dữ liệu xuống đĩa cứng sử dụng rio (stream IO).

Trong trường hợp file aof, việc thực hiện ghi dữ liệu là của background threads. Toàn bộ chức năng này được code trong file bio.c (background IO?). Thiết kế background IO này khá đơn giản. Môt loạt thread sẽ chia sẻ 1 job queue và thay nhau đợi việc từ job queue. Mỗi khi có job mới, thread sẽ chạy và thực thi job được mô tả. Có 2 loại job đơn giản:

  • REDIS_BIO_CLOSE_FILE: đóng file
  • REDIS_BIO_AOF_FSYNC: thực hiện việc flush dữ liệu từ buffer của kernel xuống buffer của đĩa cứng.process -> job 1 -> job2 -> … background threads

Tạo ra các job là công việc của child process. Để thực hiện ghi dữ liệu ra đĩa cứng, redis sẽ fork ra 1 process con. Process con này sẽ tạo ra việc cho các background threads. Một đặc điểm cùa aof file đấy là dữ liệu trong các block mới sẽ không được ghi trực tiếp vào file aof hiện tại, mà sẽ được ghi vào file tạm thời. Khi việc ghi dữ liệu hoàn thành, redis mới tiến hành ghi đè file tạm lên file thật. Việc này đảm bảo trong trường hợp hệ thống có sự cố, file aof cũ vẫn được duy trì, giúp phục hồi phần nào dữ liệu.

Trong cả 2 trường hợp, redis sử dụng tính năng Copy-on-Write của linux khi fork process con, do vậy hiệu năng không vì fork process con mà suy giảm.

Đến đây, sau khi tìm hiểu về định dạng của 2 files dữ liệu cũng như phương thức ghi dữ liệu của từng loại file, ta vẫn còn những câu hỏi mở về persistent redis như sau:

  • Tần suất ghi dữ liệu là bao nhiêu?
  • Ai chịu trách nhiệm fork process con.

Thực chất redis định nghĩa 1 giá trị gọi là tần số ghi: REDIS_DEFAULT_HZ với giá trị mặc định là 10 (redis.h). Như vậy trong 1s, redis sẽ thực hiện 10 lần việc gọi hàm fork. Toàn bộ thao tác ghi dữ liệu redis và thao tác với các key hết hạn được thực hiện bởi 1 hệ thống các “cron”. Hàm cron thực hiện việc validate các key là: databaseCron. Hàm cron thực hiện ghi dữ liệu là serverCron. Hàm serverCron sẽ được gọi theo cơ chế bất đồng bộ (dùng thư viện bất đồng bộ của chính redis) với tần số 1/1000s. Với REDIS_DEFAULT_HZ là 10, cứ 100 lần gọi, serverCron sẽ thực hiện fork child process 1 lần để ghi dữ liệu xuống bộ đĩa cứng.

5. Tại sao phải fsync

Đến đây chắc bạn đã hiểu phần nào về cơ chế persistent của redis. Tuy nhiên ta vẫn còn 1 câu hỏi nhỏ khá thú vị: tại sao phải flush liên tục như vậy (100ms / lần)? Tại sao không chỉ dùng hàm write/read của kernel và mặc định việc ghi dữ liệu xuống đĩa cứng cho kernel. Để trả lời câu hỏi nhỏ này, ta cần hiểu mối liên quan giữa OS – đĩa cứng – buffer của tầng ứng dụng.

Về mặt trực quan tra có mô hình như sau:

buffer ---(Write) --| (kernel buffer) ---> hard disk buffer ---> đĩa từ.

Một thao tác ghi dùng write/read api của kernel sẽ copy dữ liệu từ buffer tần ứng dụng xuống buffer của kernel. Đây là thao tác cơ bản của write api. Tại buffer của kernel, kernel có toàn quyền quyết định với buffer này như: khi nào ghi, ghi bao nhiêu bytes… Khi kernel ghi dữ liệu (sử dụng các hàm IO của đĩa), dữ liệu sẽ được ghi xuống hard disk buffer và được schedule ghi xuống đĩa từ. Do vậy nếu tại tầng kernel hệ thống gặp sự cố, dữ liệu vẫn có thể bị mất dù rằng write thành công (và tầng ứng dụng không có cách nào biết write không thành công). Bằng việc định kỳ gọi fsync, ứng dụng có thể thoát khỏi sử quản lý của kernel, ghi thằng dữ liệu đang có ở buffer xuống hard disk buffer. Bằng việc gọi fsync, ta tránh khỏi được rủi ro mất dữ liệu do đổ vớ ở tầng ứng dụng. Tất nhiên dữ liệu hard disk vẫn chưa hoàn toàn an toàn (ví dụ trường hợp đĩa cứng bị hỏng).

Đây là cách làm chung của các hệ thống cơ sở dữ liệu RDMS hiện hành.

6. Kết luận

Bài viết giới thiệu khái quát các module redis, đồng thời trình bày cụ thể cơ chế ghi dữ liệu của redis. Bài viết cũng làm rõ hơn ý nghĩa của fsync cũng như quy trình ghi dữ liệu của hệ điều hành. Hy vọng qua bài viết, người đọc hiểu phần nào cơ chế, hành vi của redis, qua đó sử dụng công cụ này hiệu quả hơn.

Link gốc: http://ktmt.github.io/blog/2013/07/02/tim-hieu-redis/

Running Redis as a Windows Service

Introduction

In this article I will show how to run the Window version of Redis Server, or other executables, as a Windows service.

Here at CodeProject, we use Redis as a distributed cache. We store massive amounts of information such as Articles, Forum Messages and retrieve these items from the cache rather than the database.

Redis alllows us to store and retrieve full documents, rather than querying SQL for the various pieces and composing and formatting the document on each request. This is possible on our site because most of the information is read a lot more than it is written, and some information, such as number of views, can be a little stale.

While in production we run Redis on a Linux server, in out development environment we are running the Microsoft port of Redis on a Window 7 desktop. The problem is that Redis is not designed to be run as a Windows Service. This means that someone had to logon and run the Redis executable. This was fine when Chris was the only one logging into the server, but last week, I had to connect to install a copy of our Search Server for development purposes. Needless to say, this logged Chris off, and killed the Redis Server. I restarted it under my session.

Back on my machine, I am fixing a subtle caching bug, and when I start testing, the code is acting like there is a connection failure to the Redis Server. As the code I am changing is related to the detection of problems with the connection to Redis, I spin my wheels for half an hour or so before I realize that Chris has remoted into the server, killing my session and the Redis Server.

Both Chris and I tried a number of recommended methods for running Redis as a Windows Service, without any success. Having written several Windows Services to support various CodeProject processes, I decided to write a utility which would allow us to install and run an exe as a Windows Service. This utility is called Exe2Srvc.

Using Exe2Srvc

Exe2Srv is a program that can be run as either a console application or installed as a Windows Service. This application reads the path to an executable, and the command line arguments from it .config file and then starts a the executable in a new Process.

The simplest way to use the executable is to

  1. copy the files in the binfiles download, or from bin/Release from the compiled source, into the directory containing the executable you wish to run as a Service.
  2. Edit the “Cmd” and “CmdArgs” values in the Exe2Srvc.exe.config to contain the full path to the executable, and the command line arguments required.
  3. Run the Install.bat file from a ‘Run as Administrator’ command shell.
  4. Use the Service Manager to:
    • set the start mode to Automatic
    • set the Recovery options. I usually set them to “Restart Service”.
    • start the Service.

For my tests, I downloaded Redis from Nuget, and copied the files from the /packages/Redis-64.2.6.12.1/tools under the solution directory to C:/Redis. I then copied the files fromExe2Srvc\bin\Release to the same directory.

The Exe2Srvc.exe.config file contains:

<?xml version="1.0" encoding="utf-8" />
<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.1" />
    </startup>
    <appSettings>
        <!-- set Cmd to the executable file name. -->
        <!-- set CmdArg to any command arguments required. -->
        <add key="Cmd"      value="c:\Redis\redis-server.exe"/>
        <add key="CmdArgs"  value="c:\Redis\redis.conf --port 6379"/> 
    </appSettings>
</configuration>

If the path to the executable includes spaces, then the path needs to be in quotes. This can be accomplished by enclosing the double-quoted string in single-quotes as shown below.

<?xml version="1.0" encoding="utf-8" />
<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.1" />
    </startup>
    <appSettings>
        <!-- set Cmd to the executable file name. -->
        <!-- set CmdArg to any command arguments required. -->
        <add key="Cmd"      value='"c:Program Files\Redis\redis-server.exe"'/>
        <add key="CmdArgs"  value="c:\Redis\redis.conf --port 6379"/> 
    </appSettings>
</configuration>

The Install.bat file, which must be run as an Administrator, uses the SC command to install the Service.

sc create Redis binpath= "c:\Redis\Exe2Srvc.exe"

Then I use the Services Manager to configure the Service and start it.

Service Manager

Double Click on the Redis service to open the properties editor and select the Recovery tab. Set the options to restart the service on errors as shown.

 Service Manager

Select the General tab.

  • Set the Startup Type to Automatic,
  • click the Start button
  • and click the OK button.
Service Manager

Redis is now installed and running as a Windows Service. Use the redis-cli.exe to test the connection.

How Exe2Srvc works

Creating a console application that can be run as a Windows Service used a technique shown to me by Steve Smith and based on an article Run Windows Service as a console program by Einar Egilsson.

Basically you create a console application, and change the Program class to derive from ServiceBase. In the Main, the Environment.UserInteractive property is used to determine whether the program is being run from the command line, or run as a Service.

The required command and commandline parameters are read from the LoadConfiguration method.

/// <summary>
/// Loads the configuration parameters from the application config file
/// </summary>
private void LoadConfiguration
{
    // Load the executable filename and command arguements from config file.
    _cmd     = ConfigurationManager.AppSettings["Cmd"];
    _cmdArgs = ConfigurationManager.AppSettings["CmdArgs"];
 
    if (string.IsNullOrWhiteSpace(_cmd))
        throw new Exception("The appsetting 'Cmd' was not defined in the config file.");
}

This is called from the OnStart method which is called from Main when started as a console application, or by theService infrastructure when run as a Service. OnStart runs the executable in a new Process as shown below.

/// <summary>
/// When implemented in a derived class, executes when a Start command is sent to the
/// service by the Service Control Manager (SCM) or when the operating system starts
/// (for a service that starts automatically).
/// Specifies actions to take when the service starts.
/// </summary>
/// <param name="args";>
/// Data passed by the start command.
/// </param>
protected override void OnStart(string[] args)
{
    if (Environment.UserInteractive)
    {
        string message = String.Format"Starting {0} at {1}.", _serviceName, DateTime.Now);
         Console.WriteLine(message);
    }
 
    // loading the configuration file info here allows the service to be stopped,
    // the configuration modified, and the service restarted.
    LoadConfiguration();
 
     // Start the executable.
      ProcessStartInfo procInfo = new ProcessStartInfo(_cmd);
      procInfo.UseShellExecute = false;
 
       if (!string.IsNullOrWhiteSpace(_cmdArgs))
           procInfo.Arguments = _cmdArgs;
 
       _process = Process.Start(procInfo);
 }
 

When the Service is stopped, the OnStop method is called. This kills the Process, waits for it to terminate, and disposes of the Process. Make sure you wait for the Process to terminate, failure to do so will result in improperly stopped Services that are difficult to remove and fix.

/// <summary>
/// When implemented in a derived class, executes when a Stop command is sent to the service
/// by the Service Control Manager (SCM). Specifies actions to take when a service stops running.
/// </summary>
/// <remarks>Stops the background tasks.</remarks>
protected override void OnStop()
{
    if (Environment.UserInteractive)
    {
        string message = String.Format("Stopping {0} at {1}.", _serviceName, DateTime.Now);
         Console.WriteLine(message);
     }
           
     // Kill the process
     if (_process != null)
     {
         _process.Kill();
         _process.WaitForExit();
         _process.Dispose();
         _process = null;
      }
}

Points of Interest

I’ve attempted to make this as simple and flexible as possible, but it only meets my original requirement of being able to run the Windows version of Redis as a Windows Server. If you have additional requirement, feel free to modify the code to match your needs.

Link gốc:  http://www.codeproject.com/Articles/715967/Running-Redis-as-a-Windows-Service

Distributed Caching using Redis Server with .NET/C# Client

Introduction

In this article, I would like to describe my experience with installing and configuring Redis server in most compact way. Also, I would like to do a brief overview of usage Redis hashes and lists in .NET/C# client.

In this article:

Background

Redis is one of the fastest and most feature-rich in-memory key value data stores.

Disadvantages

  • No local data buffering (like synchronized local data buffering on the Azure caching)
  • No full clusterization support yet (expected in the end of this year)

Advantages

  • Easy configuration
  • Simple usage
  • High performance
  • Support of different data types (like hashes, lists, sets, sorted sets)
  • ASP.NET session integration
  • Web UI for viewing content of the cache

In this simple demo, I’m going to demonstrate how to install and configure Redis on the server and use it from the C# code.

Redis Installation

Download binaries from https://github.com/dmajkic/redis/downloads (win32 win64 direct link) unpack archive to the application directory (e.g. C:\Program Files\Redis)

Download Redis service compiled from https://github.com/kcherenkov/redis-windows-service/downloads, then copy to the program folder (e.g. C:\Program Files\Redis. If config file is missing, download it and copy to the application directory as well. Example of the valid Redis config file is at https://raw.github.com/antirez/redis/2.6/redis.conf.

The complete set of files for Redis application is also available in zip file (x64).

When you have the full set of the application files (like it is shown on the image below),

redis application folder conten

navigate to the application directory and run the following command:

sc create %name% binpath= "\"%binpath%\" %configpath%" start= "auto" DisplayName= "Redis"

Where:

  • %name% — name of service instance, example: redis-instance;
  • %binpath% — path to this project EXE file, example: C:\Program Files\Redis\RedisService_1.1.exe;
  • %configpath% — path to redis configuration file, example: C:\Program Files\Redis\redis.conf;

Example:

sc create Redis start= auto DisplayName= Redis binpath= "\"C:\Program Files\Redis\RedisService_1.1.exe\
" \"C:\Program Files\Redis\redis.conf\""

It should look like this:

Installing redis as windows service

Make sure that you have enough privileges to start the service. After installation, check that the service was created successfully and is running now:

redis running as a windows service

Alternatively, you can use installer, created by someone (I’ve not tried): https://github.com/rgl/redis/downloads.

Redis Server Protection: Password, IP Filtering

The primary way to protect Redis server is to set IP filtering using Windows firewall or properties of the active network connection. Additional protection can be set using redis password. It needs to update the Redis config file (redis.conf) in the following way:

First, find the line:

# requirepass foobared

Remove the # symbol in the beginning and replace foobared with new password:

requirepass foobared

Then restart Redis Windows service!!!

When instantiating the client, use constructor with a password:

RedisClient client = new RedisClient(serverHost, port, redisPassword);

Redis Server Replication (master – slave configuration)

This technique allows creation copy of the server data into the synchronized copy, this means that each time when master is modified, slave server gets notification and is automatically synchronized. Mostly replication is used for read (but not write) scalability or data redundancy and for the server failover. Setup two instances of Redis (two services on the same or different servers), then configure one of them as slave. To make Redis server instance to be slave of another server, change the config file in this way:

Find the line below:

# slaveof <masterip> <masterport>

replace with:

slaveof 192.168.1.1 6379

(specify real IP of the master server, and port in case you customized it). If master is configured to require password (authentication), change redis.conf as it is shown below, find line:

# masterauth <master-password>

remove the # symbol in the beginning and replace <master-password> with master password, to be like that:

masterauth mastpassword

Now this Redis instance can be used as a readonly synchronized copy of the master server.

Using Redis Cache from the C# Code

To use Redis in C# run the Manage NuGet packages addon, find ServiceStack.Redis pack, and install it.

Sample of using Set/Get methods directly from the instantiated client:

string host = "localhost";
string elementKey = "testKeyRedis";

using (RedisClient redisClient = new RedisClient(host))
{
      if (redisClient.Get<string>(elementKey) == null)
      {
           // adding delay to see the difference
           Thread.Sleep(5000); 
           // save value in cache
           redisClient.Set(elementKey, "some cached value");
      }
      // get value from the cache by key
      message = "Item value is: " + redisClient.Get<string>("some cached value");
 }

Typed entity sets are more interesting and practical, because they operate with exact types of objects. In the code sample below, there are two classes defined Phone, and Person – owner of the phone. Each phone instance has a reference to the owner. This code demonstrates how we can add, remove or find items in the cache by criteria:

public class Phone
{
   public int Id { get; set; }
   public string Model { get; set; }
   public string Manufacturer { get; set; }
   public Person Owner { get; set; }
}

public class Person
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string Surname { get; set; }
    public int Age { get; set; }
    public string Profession { get; set; }
}

using (RedisClient redisClient = new RedisClient(host))
{
     IRedisTypedClient<phone> phones = redisClient.As<phone>();
     Phone phoneFive = phones.GetValue("5");
     if (phoneFive == null)
     {
          // make a small delay
          Thread.Sleep(5000);
          // creating a new Phone entry
          phoneFive = new Phone
          {
               Id = 5,
               Manufacturer = "Motorolla",
               Model = "xxxxx",
               Owner = new Person
               {
                    Id = 1,
                    Age = 90,
                    Name = "OldOne",
                    Profession = "sportsmen",
                    Surname = "OldManSurname"
               }
          };
          // adding Entry to the typed entity set
          phones.SetEntry(phoneFive.Id.ToString(), phoneFive);
     }
     message = "Phone model is " + phoneFive.Manufacturer;
     message += "Phone Owner Name is: " + phoneFive.Owner.Name;
}

In the example above, we instantiate the typed client IRedisTypedClient, which works with specific type of cached objects: Phone type.

ASP.NET Session State with Redis

To configure ASP.NET session state with redis provider, add a new file to your web project, namedRedisSessionStateProvider.cs, copy code from https://github.com/chadman/redis-service-provider/raw/master/RedisProvider/SessionProvider/RedisSessionProvider.cs, then add or change the following section in the configuration file (sessionState tag has to be inside system.web tag), or you can download attached sources and copy code.

<sessionstate timeout="1" mode="Custom" 
customprovider="RedisSessionStateProvider" cookieless="false">
      <providers>
        <add name="RedisSessionStateProvider" writeexceptionstoeventlog="false" 
        type="RedisProvider.SessionProvider.CustomServiceProvider" 
        server="localhost" port="6379" password="pasword">
      </add> </providers>
</sessionstate>

NOTE, that password is optional, based on the server authentication. It must be replaced with real value, or removed, if Redis server doesn’t require authentication. server attribute and port also have to be replaced according to concrete values (default port is 6379). Then in the project, you can use the session state:

// in the Global.asax
public class MvcApplication1 : System.Web.HttpApplication
{
    protected void Application_Start()
    {
        //....
    }

    protected void Session_Start()
    {
        Session["testRedisSession"] = "Message from the redis ression";
    }
}

In the Home controller:

public class HomeController : Controller
{
    public ActionResult Index()
    {
       //...
       ViewBag.Message = Session["testRedisSession"];
       return View();
    }
//...
}

Result:

redis aspnet session state

ASP.NET output cache provider with redis can be configured in the similar way.

Redis Sets and Lists

The major note is that Redis lists implement IList<T> while Redis sets implement ICollection<T>. Let’s see how we can use them.

Lists are mostly used when it needs to separate different categories of objects of the same type. For example, we have “most selling phones” and “old collection” two lists of phones:

string host = "localhost";
using (var redisClient = new RedisClient(host))
{
    //Create a 'strongly-typed' API that makes all Redis Value operations to apply against Phones
    IRedisTypedClient<phone> redis = redisClient.As<phone>();

    IRedisList<phone> mostSelling = redis.Lists["urn:phones:mostselling"];
    IRedisList<phone> oldCollection = redis.Lists["urn:phones:oldcollection"];

    Person phonesOwner = new Person
        {
            Id = 7,
            Age = 90,
            Name = "OldOne",
            Profession = "sportsmen",
            Surname = "OldManSurname"
        };
                
    // adding new items to the list
    mostSelling.Add(new Phone
            {
                Id = 5,
                Manufacturer = "Sony",
                Model = "768564564566",
                Owner = phonesOwner
            });

    mostSelling.Add(new Phone
            {
                Id = 8,
                Manufacturer = "Motorolla",
                Model = "324557546754",
                Owner = phonesOwner
            });

    var upgradedPhone  = new Phone
    {
        Id = 3,
        Manufacturer = "LG",
        Model = "634563456",
        Owner = phonesOwner
    };

    mostSelling.Add(upgradedPhone);

    // remove item from the list
    oldCollection.Remove(upgradedPhone);

    // find objects in the cache
    IEnumerable<phone> LGPhones = mostSelling.Where(ph => ph.Manufacturer == "LG");

    // find specific
    Phone singleElement = mostSelling.FirstOrDefault(ph => ph.Id == 8);

    //reset sequence and delete all lists
    redis.SetSequence(0);
    redisClient.Remove("urn:phones:mostselling");
    redisClient.Remove("urn:phones:oldcollection");
}

Redis sets are useful when it needs to store associated sets of data and gather statistical information, for example answer -> queustion, votes for an answer or question. Let’s say that we have questions and answers, it needs to store them in the cache for better performance. Using Redis, we can do it this way:

/// <summary>
/// Gets or sets the Redis Manager. The built-in IoC used with ServiceStack autowires this property.
/// </summary>
IRedisClientsManager RedisManager { get; set; }
/// <summary>
/// Delete question by performing compensating actions to 
/// StoreQuestion() to keep the datastore in a consistent state
/// </summary>
/// <param name="questionId">
public void DeleteQuestion(long questionId)
{
    using (var redis = RedisManager.GetClient())
    {
        var redisQuestions = redis.As<question>();

        var question = redisQuestions.GetById(questionId);
        if (question == null) return;
                
        //decrement score in tags list
        question.Tags.ForEach(tag => redis.IncrementItemInSortedSet("urn:tags", tag, -1));

        //remove all related answers
        redisQuestions.DeleteRelatedEntities<answer>(questionId);

        //remove this question from user index
        redis.RemoveItemFromSet("urn:user>q:" + question.UserId, questionId.ToString());

        //remove tag => questions index for each tag
        question.Tags.ForEach("urn:tags>q:" + tag.ToLower(), questionId.ToString()));

        redisQuestions.DeleteById(questionId);
    }
}

public void StoreQuestion(Question question)
{
    using (var redis = RedisManager.GetClient())
    {
        var redisQuestions = redis.As<question>();

        if (question.Tags == null) question.Tags = new List<string>();
        if (question.Id == default(long))
        {
            question.Id = redisQuestions.GetNextSequence();
            question.CreatedDate = DateTime.UtcNow;

            //Increment the popularity for each new question tag
            question.Tags.ForEach(tag => redis.IncrementItemInSortedSet("urn:tags", tag, 1));
        }

        redisQuestions.Store(question);
        redisQuestions.AddToRecentsList(question);
        redis.AddItemToSet("urn:user>q:" + question.UserId, question.Id.ToString());

        //Usage of tags - Populate tag => questions index for each tag
        question.Tags.ForEach(tag => redis.AddItemToSet
        ("urn:tags>q:" + tag.ToLower(), question.Id.ToString()));
    }
}

/// <summary>
/// Delete Answer by performing compensating actions to 
/// StoreAnswer() to keep the datastore in a consistent state
/// </summary>
/// <param name="questionId">
/// <param name="answerId">
public void DeleteAnswer(long questionId, long answerId)
{
    using (var redis = RedisManager.GetClient())
    {
        var answer = redis.As<question>().GetRelatedEntities<answer>
        (questionId).FirstOrDefault(x => x.Id == answerId);
        if (answer == null) return;
                
        redis.As<question>().DeleteRelatedEntity<answer>(questionId, answerId);
                
        //remove user => answer index
        redis.RemoveItemFromSet("urn:user>a:" + answer.UserId, answerId.ToString());
    }
}

public void StoreAnswer(Answer answer)
{
    using (var redis = RedisManager.GetClient())
    {
        if (answer.Id == default(long))
        {
            answer.Id = redis.As<answer>().GetNextSequence();
            answer.CreatedDate = DateTime.UtcNow;
        }

        //Store as a 'Related Answer' to the parent Question
        redis.As<question>().StoreRelatedEntities(answer.QuestionId, answer);
        //Populate user => answer index
        redis.AddItemToSet("urn:user>a:" + answer.UserId, answer.Id.ToString());
    }
}

public List<answer> GetAnswersForQuestion(long questionId)
{
    using (var redis = RedisManager.GetClient())
    {
        return redis.As<question>().GetRelatedEntities<answer>(questionId);
    }
}

public void VoteQuestionUp(long userId, long questionId)
{
    //Populate Question => User and User => Question set indexes in a single transaction
    RedisManager.ExecTrans(trans =>
    {
        //Register upvote against question and remove any downvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:q>user+:" + questionId, userId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:q>user-:" + questionId, userId.ToString()));

        //Register upvote against user and remove any downvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:user>q+:" + userId, questionId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:user>q-:" + userId, questionId.ToString()));
    });
}

public void VoteQuestionDown(long userId, long questionId)
{
    //Populate Question => User and User => Question set indexes in a single transaction
    RedisManager.ExecTrans(trans =>
    {
        //Register downvote against question and remove any upvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:q>user-:" + questionId, userId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:q>user+:" + questionId, userId.ToString()));

        //Register downvote against user and remove any upvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet"urn:user>q-:" + userId, questionId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:user>q+:" + userId, questionId.ToString()));
    });
}

public void VoteAnswerUp(long userId, long answerId)
{
    //Populate Question => User and User => Question set indexes in a single transaction
    RedisManager.ExecTrans(trans =>
    {
        //Register upvote against answer and remove any downvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:a>user+:" + answerId, userId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:a>user-:" + answerId, userId.ToString()));

        //Register upvote against user and remove any downvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:user>a+:" + userId, answerId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:user>a-:" + userId, answerId.ToString()));
    });
}

public void VoteAnswerDown(long userId, long answerId)
{
    //Populate Question => User and User => Question set indexes in a single transaction
    RedisManager.ExecTrans(trans =>
    {
        //Register downvote against answer and remove any upvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:a>user-:" + answerId, userId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:a>user+:" + answerId, userId.ToString()));

        //Register downvote against user and remove any upvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:user>a-:" + userId, answerId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:user>a+:" + userId, answerId.ToString()));
    });
}

public QuestionResult GetQuestion(long questionId)
{
    var question = RedisManager.ExecAs<question>
    (redisQuestions => redisQuestions.GetById(questionId));
    if (question == null) return null;

    var result = ToQuestionResults(new[] { question })[0];
    var answers = GetAnswersForQuestion(questionId);
    var uniqueUserIds = answers.ConvertAll(x => x.UserId).ToHashSet();
    var usersMap = GetUsersByIds(uniqueUserIds).ToDictionary(x => x.Id);

    result.Answers = answers.ConvertAll(answer =>
        new AnswerResult { Answer = answer, User = usersMap[answer.UserId] });

    return result;
}

public List<user> GetUsersByIds(IEnumerable<long> userIds)
{
    return RedisManager.ExecAs<user>(redisUsers => redisUsers.GetByIds(userIds)).ToList();
}

public QuestionStat GetQuestionStats(long questionId)
{
    using (var redis = RedisManager.GetReadOnlyClient())
    {
        var result = new QuestionStat
        {
            VotesUpCount = redis.GetSetCount("urn:q>user+:" +questionId),
            VotesDownCount = redis.GetSetCount("urn:q>user-:" + questionId)
        };
        result.VotesTotal = result.VotesUpCount - result.VotesDownCount;
        return result;
    }
}

public List<tag> GetTagsByPopularity(int skip, int take)
{
    using (var redis = RedisManager.GetReadOnlyClient())
    {
        var tagEntries = redis.GetRangeWithScoresFromSortedSetDesc("urn:tags", skip, take);
        var tags = tagEntries.ConvertAll(kvp => new Tag { Name = kvp.Key, Score = (int)kvp.Value });
        return tags;
    }
}

public SiteStats GetSiteStats()
{
    using (var redis = RedisManager.GetClient())
    {
        return new SiteStats
        {
            QuestionsCount = redis.As<question>().TypeIdsSet.Count,
            AnswersCount = redis.As<answer>().TypeIdsSet.Count,
            TopTags = GetTagsByPopularity(0, 10)
        };
    }
}

Attached Sources Description

List of included packages is in the packages.config,
Funq IoC configuration, registering types and current controller factory – are in the Global.asax (property dependency injection)
Usage of a simple client – in the home controller
Usage of IoC based cache in the Question and Answer controllers, and Global.asax application file. To see how it works, you can run the project, and open in the browser following URL:http://localhost:37447/Question/GetQuestions?tag=test .
You can play with tags, like test3, test11, test2, etc.
Redis Cache configuration – in the web config (<system.web><sessionState> section) and in theRedisSessionStateProvider.cs file.
There are a lot of TODOs in the MVC project, so if you want to improve/continue please update it, and upload.

I would much appreciate if someone could help build MVC application with simple UI, using Redis (with Funq IoC) cache. Funq IoC is already configured, example of the usage is in the Question controller.

NOTE: Samples were partially taken from the “ServiceStack.Examples-master” solution

Conclusion. Optimization Caching in Application with Fast Local Cache

Since Redis doesn’t store data locally (no local replication), it might make sence to optimize performance by storing some light or user – dependent objects in the local cache (to skip serialization to string and client – server data transfering). For example, in the web application, it is better to use ‘System.Runtime.Caching.ObjectCache‘ for light objects, which are user dependent and used frequently by the application. Otherwise, when object has common usage, large size it must be saved in the distributed Redis cache. Example of the user dependent objects – profile information, personalization information. Common objects – localization data, information shared between different users, etc.

LINK gốc:  http://www.codeproject.com/Articles/636730/Distributed-Caching-using-Redis

Distributed Caching using Redis Server with .NET/C# Client

Introduction

In this article, I would like to describe my experience with installing and configuring Redis server in most compact way. Also, I would like to do a brief overview of usage Redis hashes and lists in .NET/C# client.

In this article:

Background

Redis is one of the fastest and most feature-rich in-memory key value data stores.

Disadvantages

  • No local data buffering (like synchronized local data buffering on the Azure caching)
  • No full clusterization support yet (expected in the end of this year)

Advantages

  • Easy configuration
  • Simple usage
  • High performance
  • Support of different data types (like hashes, lists, sets, sorted sets)
  • ASP.NET session integration
  • Web UI for viewing content of the cache

In this simple demo, I’m going to demonstrate how to install and configure Redis on the server and use it from the C# code.

Redis Installation

Download binaries from https://github.com/dmajkic/redis/downloads (win32 win64 direct link) unpack archive to the application directory (e.g. C:\Program Files\Redis)

Download Redis service compiled from https://github.com/kcherenkov/redis-windows-service/downloads, then copy to the program folder (e.g. C:\Program Files\Redis. If config file is missing, download it and copy to the application directory as well. Example of the valid Redis config file is at https://raw.github.com/antirez/redis/2.6/redis.conf.

The complete set of files for Redis application is also available in zip file (x64).

When you have the full set of the application files (like it is shown on the image below),

redis application folder conten

navigate to the application directory and run the following command:

 Collapse | Copy Code
sc create %name% binpath= "\"%binpath%\" %configpath%" start= "auto" DisplayName= "Redis"

Where:

  • %name% — name of service instance, example: redis-instance;
  • %binpath% — path to this project EXE file, example: C:\Program Files\Redis\RedisService_1.1.exe;
  • %configpath% — path to redis configuration file, example: C:\Program Files\Redis\redis.conf;

Example:

 Collapse | Copy Code
sc create Redis start= auto DisplayName= Redis binpath= "\"C:\Program Files\Redis\RedisService_1.1.exe\
" \"C:\Program Files\Redis\redis.conf\""

It should look like this:

Installing redis as windows service

Make sure that you have enough privileges to start the service. After installation, check that the service was created successfully and is running now:

redis running as a windows service

Alternatively, you can use installer, created by someone (I’ve not tried): https://github.com/rgl/redis/downloads.

Redis Server Protection: Password, IP Filtering

The primary way to protect Redis server is to set IP filtering using Windows firewall or properties of the active network connection. Additional protection can be set using redis password. It needs to update the Redis config file (redis.conf) in the following way:

First, find the line:

 Collapse | Copy Code
# requirepass foobared

Remove the # symbol in the beginning and replace foobared with new password:

 Collapse | Copy Code
requirepass foobared

Then restart Redis Windows service!!!

When instantiating the client, use constructor with a password:

 Collapse | Copy Code
RedisClient client = new RedisClient(serverHost, port, redisPassword);

Redis Server Replication (master – slave configuration)

This technique allows creation copy of the server data into the synchronized copy, this means that each time when master is modified, slave server gets notification and is automatically synchronized. Mostly replication is used for read (but not write) scalability or data redundancy and for the server failover. Setup two instances of Redis (two services on the same or different servers), then configure one of them as slave. To make Redis server instance to be slave of another server, change the config file in this way:

Find the line below:

 Collapse | Copy Code
# slaveof <masterip> <masterport>

replace with:

 Collapse | Copy Code
slaveof 192.168.1.1 6379

(specify real IP of the master server, and port in case you customized it). If master is configured to require password (authentication), change redis.conf as it is shown below, find line:

 Collapse | Copy Code
# masterauth <master-password>

remove the # symbol in the beginning and replace <master-password> with master password, to be like that:

 Collapse | Copy Code
masterauth mastpassword

Now this Redis instance can be used as a readonly synchronized copy of the master server.

Using Redis Cache from the C# Code

To use Redis in C# run the Manage NuGet packages addon, find ServiceStack.Redis pack, and install it.

Sample of using Set/Get methods directly from the instantiated client:

 Collapse | Copy Code
string host = "localhost";
string elementKey = "testKeyRedis";

using (RedisClient redisClient = new RedisClient(host))
{
      if (redisClient.Get<string>(elementKey) == null)
      {
           // adding delay to see the difference
           Thread.Sleep(5000); 
           // save value in cache
           redisClient.Set(elementKey, "some cached value");
      }
      // get value from the cache by key
      message = "Item value is: " + redisClient.Get<string>("some cached value");
 }

Typed entity sets are more interesting and practical, because they operate with exact types of objects. In the code sample below, there are two classes defined Phone, and Person – owner of the phone. Each phone instance has a reference to the owner. This code demonstrates how we can add, remove or find items in the cache by criteria:

 Collapse | Copy Code
public class Phone
{
   public int Id { get; set; }
   public string Model { get; set; }
   public string Manufacturer { get; set; }
   public Person Owner { get; set; }
}

public class Person
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string Surname { get; set; }
    public int Age { get; set; }
    public string Profession { get; set; }
}

using (RedisClient redisClient = new RedisClient(host))
{
     IRedisTypedClient<phone> phones = redisClient.As<phone>();
     Phone phoneFive = phones.GetValue("5");
     if (phoneFive == null)
     {
          // make a small delay
          Thread.Sleep(5000);
          // creating a new Phone entry
          phoneFive = new Phone
          {
               Id = 5,
               Manufacturer = "Motorolla",
               Model = "xxxxx",
               Owner = new Person
               {
                    Id = 1,
                    Age = 90,
                    Name = "OldOne",
                    Profession = "sportsmen",
                    Surname = "OldManSurname"
               }
          };
          // adding Entry to the typed entity set
          phones.SetEntry(phoneFive.Id.ToString(), phoneFive);
     }
     message = "Phone model is " + phoneFive.Manufacturer;
     message += "Phone Owner Name is: " + phoneFive.Owner.Name;
}

In the example above, we instantiate the typed client IRedisTypedClient, which works with specific type of cached objects: Phone type.

ASP.NET Session State with Redis

To configure ASP.NET session state with redis provider, add a new file to your web project, namedRedisSessionStateProvider.cs, copy code from https://github.com/chadman/redis-service-provider/raw/master/RedisProvider/SessionProvider/RedisSessionProvider.cs, then add or change the following section in the configuration file (sessionState tag has to be inside system.web tag), or you can download attached sources and copy code.

 Collapse | Copy Code
<sessionstate timeout="1" mode="Custom" 
customprovider="RedisSessionStateProvider" cookieless="false">
      <providers>
        <add name="RedisSessionStateProvider" writeexceptionstoeventlog="false" 
        type="RedisProvider.SessionProvider.CustomServiceProvider" 
        server="localhost" port="6379" password="pasword">
      </add> </providers>
</sessionstate>

NOTE, that password is optional, based on the server authentication. It must be replaced with real value, or removed, if Redis server doesn’t require authentication. server attribute and port also have to be replaced according to concrete values (default port is 6379). Then in the project, you can use the session state:

 Collapse | Copy Code
// in the Global.asax
public class MvcApplication1 : System.Web.HttpApplication
{
    protected void Application_Start()
    {
        //....
    }

    protected void Session_Start()
    {
        Session["testRedisSession"] = "Message from the redis ression";
    }
}

In the Home controller:

 Collapse | Copy Code
public class HomeController : Controller
{
    public ActionResult Index()
    {
       //...
       ViewBag.Message = Session["testRedisSession"];
       return View();
    }
//...
}

Result:

redis aspnet session state

ASP.NET output cache provider with redis can be configured in the similar way.

Redis Sets and Lists

The major note is that Redis lists implement IList<T> while Redis sets implement ICollection<T>. Let’s see how we can use them.

Lists are mostly used when it needs to separate different categories of objects of the same type. For example, we have “most selling phones” and “old collection” two lists of phones:

 Collapse | Copy Code
string host = "localhost";
using (var redisClient = new RedisClient(host))
{
    //Create a 'strongly-typed' API that makes all Redis Value operations to apply against Phones
    IRedisTypedClient<phone> redis = redisClient.As<phone>();

    IRedisList<phone> mostSelling = redis.Lists["urn:phones:mostselling"];
    IRedisList<phone> oldCollection = redis.Lists["urn:phones:oldcollection"];

    Person phonesOwner = new Person
        {
            Id = 7,
            Age = 90,
            Name = "OldOne",
            Profession = "sportsmen",
            Surname = "OldManSurname"
        };

    // adding new items to the list
    mostSelling.Add(new Phone
            {
                Id = 5,
                Manufacturer = "Sony",
                Model = "768564564566",
                Owner = phonesOwner
            });

    mostSelling.Add(new Phone
            {
                Id = 8,
                Manufacturer = "Motorolla",
                Model = "324557546754",
                Owner = phonesOwner
            });

    var upgradedPhone  = new Phone
    {
        Id = 3,
        Manufacturer = "LG",
        Model = "634563456",
        Owner = phonesOwner
    };

    mostSelling.Add(upgradedPhone);

    // remove item from the list
    oldCollection.Remove(upgradedPhone);

    // find objects in the cache
    IEnumerable<phone> LGPhones = mostSelling.Where(ph => ph.Manufacturer == "LG");

    // find specific
    Phone singleElement = mostSelling.FirstOrDefault(ph => ph.Id == 8);

    //reset sequence and delete all lists
    redis.SetSequence(0);
    redisClient.Remove("urn:phones:mostselling");
    redisClient.Remove("urn:phones:oldcollection");
}

Redis sets are useful when it needs to store associated sets of data and gather statistical information, for example answer -> queustion, votes for an answer or question. Let’s say that we have questions and answers, it needs to store them in the cache for better performance. Using Redis, we can do it this way:

 Collapse | Copy Code
/// <summary>
/// Gets or sets the Redis Manager. The built-in IoC used with ServiceStack autowires this property.
/// </summary>
IRedisClientsManager RedisManager { get; set; }
/// <summary>
/// Delete question by performing compensating actions to 
/// StoreQuestion() to keep the datastore in a consistent state
/// </summary>
/// <param name="questionId">
public void DeleteQuestion(long questionId)
{
    using (var redis = RedisManager.GetClient())
    {
        var redisQuestions = redis.As<question>();

        var question = redisQuestions.GetById(questionId);
        if (question == null) return;

        //decrement score in tags list
        question.Tags.ForEach(tag => redis.IncrementItemInSortedSet("urn:tags", tag, -1));

        //remove all related answers
        redisQuestions.DeleteRelatedEntities<answer>(questionId);

        //remove this question from user index
        redis.RemoveItemFromSet("urn:user>q:" + question.UserId, questionId.ToString());

        //remove tag => questions index for each tag
        question.Tags.ForEach("urn:tags>q:" + tag.ToLower(), questionId.ToString()));

        redisQuestions.DeleteById(questionId);
    }
}

public void StoreQuestion(Question question)
{
    using (var redis = RedisManager.GetClient())
    {
        var redisQuestions = redis.As<question>();

        if (question.Tags == null) question.Tags = new List<string>();
        if (question.Id == default(long))
        {
            question.Id = redisQuestions.GetNextSequence();
            question.CreatedDate = DateTime.UtcNow;

            //Increment the popularity for each new question tag
            question.Tags.ForEach(tag => redis.IncrementItemInSortedSet("urn:tags", tag, 1));
        }

        redisQuestions.Store(question);
        redisQuestions.AddToRecentsList(question);
        redis.AddItemToSet("urn:user>q:" + question.UserId, question.Id.ToString());

        //Usage of tags - Populate tag => questions index for each tag
        question.Tags.ForEach(tag => redis.AddItemToSet
        ("urn:tags>q:" + tag.ToLower(), question.Id.ToString()));
    }
}

/// <summary>
/// Delete Answer by performing compensating actions to 
/// StoreAnswer() to keep the datastore in a consistent state
/// </summary>
/// <param name="questionId">
/// <param name="answerId">
public void DeleteAnswer(long questionId, long answerId)
{
    using (var redis = RedisManager.GetClient())
    {
        var answer = redis.As<question>().GetRelatedEntities<answer>
        (questionId).FirstOrDefault(x => x.Id == answerId);
        if (answer == null) return;

        redis.As<question>().DeleteRelatedEntity<answer>(questionId, answerId);

        //remove user => answer index
        redis.RemoveItemFromSet("urn:user>a:" + answer.UserId, answerId.ToString());
    }
}

public void StoreAnswer(Answer answer)
{
    using (var redis = RedisManager.GetClient())
    {
        if (answer.Id == default(long))
        {
            answer.Id = redis.As<answer>().GetNextSequence();
            answer.CreatedDate = DateTime.UtcNow;
        }

        //Store as a 'Related Answer' to the parent Question
        redis.As<question>().StoreRelatedEntities(answer.QuestionId, answer);
        //Populate user => answer index
        redis.AddItemToSet("urn:user>a:" + answer.UserId, answer.Id.ToString());
    }
}

public List<answer> GetAnswersForQuestion(long questionId)
{
    using (var redis = RedisManager.GetClient())
    {
        return redis.As<question>().GetRelatedEntities<answer>(questionId);
    }
}

public void VoteQuestionUp(long userId, long questionId)
{
    //Populate Question => User and User => Question set indexes in a single transaction
    RedisManager.ExecTrans(trans =>
    {
        //Register upvote against question and remove any downvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:q>user+:" + questionId, userId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:q>user-:" + questionId, userId.ToString()));

        //Register upvote against user and remove any downvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:user>q+:" + userId, questionId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:user>q-:" + userId, questionId.ToString()));
    });
}

public void VoteQuestionDown(long userId, long questionId)
{
    //Populate Question => User and User => Question set indexes in a single transaction
    RedisManager.ExecTrans(trans =>
    {
        //Register downvote against question and remove any upvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:q>user-:" + questionId, userId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:q>user+:" + questionId, userId.ToString()));

        //Register downvote against user and remove any upvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet"urn:user>q-:" + userId, questionId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:user>q+:" + userId, questionId.ToString()));
    });
}

public void VoteAnswerUp(long userId, long answerId)
{
    //Populate Question => User and User => Question set indexes in a single transaction
    RedisManager.ExecTrans(trans =>
    {
        //Register upvote against answer and remove any downvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:a>user+:" + answerId, userId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:a>user-:" + answerId, userId.ToString()));

        //Register upvote against user and remove any downvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:user>a+:" + userId, answerId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:user>a-:" + userId, answerId.ToString()));
    });
}

public void VoteAnswerDown(long userId, long answerId)
{
    //Populate Question => User and User => Question set indexes in a single transaction
    RedisManager.ExecTrans(trans =>
    {
        //Register downvote against answer and remove any upvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:a>user-:" + answerId, userId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:a>user+:" + answerId, userId.ToString()));

        //Register downvote against user and remove any upvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:user>a-:" + userId, answerId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:user>a+:" + userId, answerId.ToString()));
    });
}

public QuestionResult GetQuestion(long questionId)
{
    var question = RedisManager.ExecAs<question>
    (redisQuestions => redisQuestions.GetById(questionId));
    if (question == null) return null;

    var result = ToQuestionResults(new[] { question })[0];
    var answers = GetAnswersForQuestion(questionId);
    var uniqueUserIds = answers.ConvertAll(x => x.UserId).ToHashSet();
    var usersMap = GetUsersByIds(uniqueUserIds).ToDictionary(x => x.Id);

    result.Answers = answers.ConvertAll(answer =>
        new AnswerResult { Answer = answer, User = usersMap[answer.UserId] });

    return result;
}

public List<user> GetUsersByIds(IEnumerable<long> userIds)
{
    return RedisManager.ExecAs<user>(redisUsers => redisUsers.GetByIds(userIds)).ToList();
}

public QuestionStat GetQuestionStats(long questionId)
{
    using (var redis = RedisManager.GetReadOnlyClient())
    {
        var result = new QuestionStat
        {
            VotesUpCount = redis.GetSetCount("urn:q>user+:" +questionId),
            VotesDownCount = redis.GetSetCount("urn:q>user-:" + questionId)
        };
        result.VotesTotal = result.VotesUpCount - result.VotesDownCount;
        return result;
    }
}

public List<tag> GetTagsByPopularity(int skip, int take)
{
    using (var redis = RedisManager.GetReadOnlyClient())
    {
        var tagEntries = redis.GetRangeWithScoresFromSortedSetDesc("urn:tags", skip, take);
        var tags = tagEntries.ConvertAll(kvp => new Tag { Name = kvp.Key, Score = (int)kvp.Value });
        return tags;
    }
}

public SiteStats GetSiteStats()
{
    using (var redis = RedisManager.GetClient())
    {
        return new SiteStats
        {
            QuestionsCount = redis.As<question>().TypeIdsSet.Count,
            AnswersCount = redis.As<answer>().TypeIdsSet.Count,
            TopTags = GetTagsByPopularity(0, 10)
        };
    }
}

Attached Sources Description

List of included packages is in the packages.config,
Funq IoC configuration, registering types and current controller factory – are in the Global.asax (property dependency injection)
Usage of a simple client – in the home controller
Usage of IoC based cache in the Question and Answer controllers, and Global.asax application file. To see how it works, you can run the project, and open in the browser following URL:http://localhost:37447/Question/GetQuestions?tag=test .
You can play with tags, like test3, test11, test2, etc.
Redis Cache configuration – in the web config (<system.web><sessionState> section) and in theRedisSessionStateProvider.cs file.
There are a lot of TODOs in the MVC project, so if you want to improve/continue please update it, and upload.

I would much appreciate if someone could help build MVC application with simple UI, using Redis (with Funq IoC) cache. Funq IoC is already configured, example of the usage is in the Question controller.

NOTE: Samples were partially taken from the “ServiceStack.Examples-master” solution

Conclusion. Optimization Caching in Application with Fast Local Cache

Since Redis doesn’t store data locally (no local replication), it might make sence to optimize performance by storing some light or user – dependent objects in the local cache (to skip serialization to string and client – server data transfering). For example, in the web application, it is better to use ‘System.Runtime.Caching.ObjectCache‘ for light objects, which are user dependent and used frequently by the application. Otherwise, when object has common usage, large size it must be saved in the distributed Redis cache. Example of the user dependent objects – profile information, personalization information. Common objects – localization data, information shared between different users, etc.

 

Link goc: http://www.codeproject.com/Articles/636730/Distributed-Caching-using-Redis