Tìm Hiểu Redis – Phần 1

1. Giới thiệu

Redis là hệ thống lưu trữ key-value với rất nhiều tính năng và được sử dụng rộng rãi. Redis nổi bật bởi việc hỗ trợ nhiều cấu trúc dữ liệu cơ bản (hash, list, set, sorted set, string), đồng thời cho phép scripting bằng ngôn ngữ lua. Bên cạnh lưu trữ key-value trên RAM với hiệu năng cao, redis còn hỗ trợ lưu trữ dữ liệu trên đĩa cứng (persistent redis) cho phép phục hồi dữ liệu khi gặp sự cố. Ngoài tính năng replicatation (sao chép giữa master-client), tính năng cluster (sao lưu master-master) cũng đang được phát triển . Để sử dụng một cách hiệu quả những tính năng redis hỗ trợ cũng như vận hành redis với hiệu suất cao nhất thì việc am hiểu hệ thống lưu trữ này là điều không thể thiếu. Chính vì lý do này, mình quyết định tìm hiểu mã nguồn redis. Loạt bài viết về redis này tóm tắt những điều mình tìm hiểu được từ việc đọc mã nguồn của redis.

2. Khái quát

Bạn có thể clone mã nguồn redis về máy tính mình bằng câu lệnh dưới đây:

git clone https://github.com/antirez/redis.git

Trước hết là một số thống kê nho nhỏ về redis (tại thời điểm bài viết): * Số lượng file mã nguồn: 55

ls *.c | wc -l
55
  • Số lượng file header: 30
    ls *.h | wc -l
    30
  • Tổng số dòng code: 43829
    wc -l *.[ch]
    341    adlist.c     197   pqsort.c            228   sha1.c          810   dict.c
    93     adlist.h     40    pqsort.h            17    sha1.h          173   dict.h
    435    ae.c         359   pubsub.c            169   slowlog.c       124   endianconv.c
    130    ae_epoll.c   93    rand.c              47    slowlog.h       64    endianconv.h
    315    ae_evport.c  38    rand.h              50    solarisfixes.h  52    fmacros.h
    118    ae.h         1230  rdb.c               530   sort.c          759   help.h
    132    ae_kqueue.c  114   rdb.h               144   syncio.c        483   intset.c
    99     ae_select.c  683   redis-benchmark.c   57    testhelp.h      50    intset.h
    441    anet.c       3008  redis.c             761   t_hash.c        295   lzf_c.c
    60     anet.h       218   redis-check-aof.c   1149  t_list.c        150   lzf_d.c
    1178   aof.c        768   redis-check-dump.c  913   t_set.c         100   lzf.h
    47     asciilogo.h  1556  redis-cli.c         459   t_string.c      159   lzfP.h
    220    bio.c        1517  redis.h             2205  t_zset.c        279   memtest.c
    41     bio.h        52    release.c           520   util.c          323   multi.c
    412    bitops.c     3     release.h           41    util.h          1444  networking.c
    2866   cluster.c    1658  replication.c       1     version.h       128   notify.c
    1726   config.c     198   rio.c               1534  ziplist.c       580   object.c
    195    config.h     104   rio.h               46    ziplist.h
    88     crc16.c      1065  scripting.c         467   zipmap.c
    191    crc64.c      732   sds.c               49    zipmap.h
    8      crc64.h      99    sds.h               351   zmalloc.c
    815    db.c         3160  sentinel.c          85    zmalloc.h
    929    debug.c      261   setproctitle.c
    43829  total

Một số thư viện được sử dụng: jemalloclinenoiselua

3. Các modules

Redis bao gồm các module sau:

  • Framework hỗ trợ xử lý bất đồng bộ, networking: ae, anet
  • Mô tả dữ liệu: sds.c, t_hash.c, t_list.c, t_string.c, t_zset.c, object.c, notify.c (pub-sub)
  • Lưu trữ dữ liệu, cơ sở dữ liệu: db.c, dict.c, ziplist.c, zipmap.c, adlist.c
  • Module hỗ trợ IO/persistent redis: rdb.c, aof.c, bio.c, rio.c
  • Utilities: crc16.c, crc64.c, pqsort.c, lzf_c.c, lzf_d.c

Mình sẽ lần lượt giới thiệu các modules trong các bài viết sau. Ở bài viết này, mình sẽ tập trung vào module IO/persistent redis.

4. Persistent redis

Bên cạnh việc lưu key-value trên bộ nhớ RAM, Redis có 2 background threads chuyên làm nhiệm vụ định kỳ ghi dữ liệu lên đĩa cứng.

Có 2 loại file được ghi xuống đĩa cứng:

  • RDB
  • AOF

RDB lưu dữ liệu dưới dạng đã mã hóa. AOF lưu lại toàn bộ dưới liệu dưới dạng command, giống như command mà redis client gửi đến server để thao tác bằng cách ghi đè xuống cuối file.

File rdb có thể coi là một snapshot của cơ sở dữ liệu tại một thời điểm nhất định. File dữ liệu này được dùng với 2 mục đích

  • Cho phép redis có thể phục hồi lại dữ liệu trên memory bằng việc đọc file
  • Bản thân dữ liệu được ghi ra file rdb sẽ được gửi đến các redis slave server, phục vụ mục đích sao lưu server.

Dữ liệu ghi ra file rdb được chỉnh sửa và mã hóa để giảm kích thước ghi trên đĩa, đồng thời tằng tốc độ replication. Cụ thể định dạng của file rdb như sau.

Với những key ngắn, việc dùng 32 bit để mô tả key là thừa thãi, do vậy redis quy định những key ngắn được mã hóa sử dụng 2 bit đầu tiên của 1 byte. Cụ thể: 00|000000 => 00, độ dài dữ liệu mô tả bởi 6 bit 01|000000 00000000 => 01, độ dài dữ liệu là 14 bit 10|000000 [số 32 bit] => 1 chuỗi độ dài 32 bit sẽ theo sau 11|000000 => obj được encode đặc biệt sẽ theo sau byte này. 6 bit sau sẽ xác định kiểu object.

Kiểu object ở đây cụ thể như sau:

rdb.c
1
2
3
4
5
6
7
/* When a length of a string object stored on disk has the first two bits
 * set, the remaining two bits specify a special encoding for the object
 * accordingly to the following defines: */
#define REDIS_RDB_ENC_INT8 0        /* 8 bit signed integer */
#define REDIS_RDB_ENC_INT16 1       /* 16 bit signed integer */
#define REDIS_RDB_ENC_INT32 2       /* 32 bit signed integer */
#define REDIS_RDB_ENC_LZF 3         /* string compressed with FASTLZ */

Với AOF file, các command sẽ được nhóm thành các block. Các block được tổ chức dưới dạng danh sách liên kết. Mỗi block có độ lớn 10MB là vì trong trường hợp redis server chịu tải cao, số lượng key được cập nhật lớn, nếu kích thước buffer lớn, việc realloc buffer dùng cho các command với tốc độ lớn không đảm bảo.

Trong trường hợp file rdb, redis fork 1 process con và thực hiện ghi dữ liệu xuống đĩa cứng sử dụng rio (stream IO).

Trong trường hợp file aof, việc thực hiện ghi dữ liệu là của background threads. Toàn bộ chức năng này được code trong file bio.c (background IO?). Thiết kế background IO này khá đơn giản. Môt loạt thread sẽ chia sẻ 1 job queue và thay nhau đợi việc từ job queue. Mỗi khi có job mới, thread sẽ chạy và thực thi job được mô tả. Có 2 loại job đơn giản:

  • REDIS_BIO_CLOSE_FILE: đóng file
  • REDIS_BIO_AOF_FSYNC: thực hiện việc flush dữ liệu từ buffer của kernel xuống buffer của đĩa cứng.process -> job 1 -> job2 -> … background threads

Tạo ra các job là công việc của child process. Để thực hiện ghi dữ liệu ra đĩa cứng, redis sẽ fork ra 1 process con. Process con này sẽ tạo ra việc cho các background threads. Một đặc điểm cùa aof file đấy là dữ liệu trong các block mới sẽ không được ghi trực tiếp vào file aof hiện tại, mà sẽ được ghi vào file tạm thời. Khi việc ghi dữ liệu hoàn thành, redis mới tiến hành ghi đè file tạm lên file thật. Việc này đảm bảo trong trường hợp hệ thống có sự cố, file aof cũ vẫn được duy trì, giúp phục hồi phần nào dữ liệu.

Trong cả 2 trường hợp, redis sử dụng tính năng Copy-on-Write của linux khi fork process con, do vậy hiệu năng không vì fork process con mà suy giảm.

Đến đây, sau khi tìm hiểu về định dạng của 2 files dữ liệu cũng như phương thức ghi dữ liệu của từng loại file, ta vẫn còn những câu hỏi mở về persistent redis như sau:

  • Tần suất ghi dữ liệu là bao nhiêu?
  • Ai chịu trách nhiệm fork process con.

Thực chất redis định nghĩa 1 giá trị gọi là tần số ghi: REDIS_DEFAULT_HZ với giá trị mặc định là 10 (redis.h). Như vậy trong 1s, redis sẽ thực hiện 10 lần việc gọi hàm fork. Toàn bộ thao tác ghi dữ liệu redis và thao tác với các key hết hạn được thực hiện bởi 1 hệ thống các “cron”. Hàm cron thực hiện việc validate các key là: databaseCron. Hàm cron thực hiện ghi dữ liệu là serverCron. Hàm serverCron sẽ được gọi theo cơ chế bất đồng bộ (dùng thư viện bất đồng bộ của chính redis) với tần số 1/1000s. Với REDIS_DEFAULT_HZ là 10, cứ 100 lần gọi, serverCron sẽ thực hiện fork child process 1 lần để ghi dữ liệu xuống bộ đĩa cứng.

5. Tại sao phải fsync

Đến đây chắc bạn đã hiểu phần nào về cơ chế persistent của redis. Tuy nhiên ta vẫn còn 1 câu hỏi nhỏ khá thú vị: tại sao phải flush liên tục như vậy (100ms / lần)? Tại sao không chỉ dùng hàm write/read của kernel và mặc định việc ghi dữ liệu xuống đĩa cứng cho kernel. Để trả lời câu hỏi nhỏ này, ta cần hiểu mối liên quan giữa OS – đĩa cứng – buffer của tầng ứng dụng.

Về mặt trực quan tra có mô hình như sau:

buffer ---(Write) --| (kernel buffer) ---> hard disk buffer ---> đĩa từ.

Một thao tác ghi dùng write/read api của kernel sẽ copy dữ liệu từ buffer tần ứng dụng xuống buffer của kernel. Đây là thao tác cơ bản của write api. Tại buffer của kernel, kernel có toàn quyền quyết định với buffer này như: khi nào ghi, ghi bao nhiêu bytes… Khi kernel ghi dữ liệu (sử dụng các hàm IO của đĩa), dữ liệu sẽ được ghi xuống hard disk buffer và được schedule ghi xuống đĩa từ. Do vậy nếu tại tầng kernel hệ thống gặp sự cố, dữ liệu vẫn có thể bị mất dù rằng write thành công (và tầng ứng dụng không có cách nào biết write không thành công). Bằng việc định kỳ gọi fsync, ứng dụng có thể thoát khỏi sử quản lý của kernel, ghi thằng dữ liệu đang có ở buffer xuống hard disk buffer. Bằng việc gọi fsync, ta tránh khỏi được rủi ro mất dữ liệu do đổ vớ ở tầng ứng dụng. Tất nhiên dữ liệu hard disk vẫn chưa hoàn toàn an toàn (ví dụ trường hợp đĩa cứng bị hỏng).

Đây là cách làm chung của các hệ thống cơ sở dữ liệu RDMS hiện hành.

6. Kết luận

Bài viết giới thiệu khái quát các module redis, đồng thời trình bày cụ thể cơ chế ghi dữ liệu của redis. Bài viết cũng làm rõ hơn ý nghĩa của fsync cũng như quy trình ghi dữ liệu của hệ điều hành. Hy vọng qua bài viết, người đọc hiểu phần nào cơ chế, hành vi của redis, qua đó sử dụng công cụ này hiệu quả hơn.

Link gốc: http://ktmt.github.io/blog/2013/07/02/tim-hieu-redis/

Advertisements

Working with NoSQL Databases

Editorial Note

This articles was originally at wiki.asp.net but has now been given a new home on CodeProject. Editing rights for this article has been set at Bronze or above, so please go in and edit and update this article to keep it fresh and relevant.

Working with NoSQL Databases

How to get started with NoSQL?

Since 2009, NoSQL databases becomes more and more popular. But why?

  • No usage of SQL, that means
    • Less complexity
    • Better portability
    • Boundlessness
    • User-friendliness
  • Most databases are Open-Source
  • Performance
  • Scalability

Famous companies like Twitter, Facebook and Amazon are using NoSQL databases.
What sorts of NoSQL databases are used today?

  • Key-Value stores
    • Easy to implement
    • Only key-value-pairs can be stored
    • Difficult to build complex data structures
  • Column stores
    • Columns don’t have to be defined in advance.
    • A row can have different numbers of cells
  • Document stores
    • Like key-value stores, but allows nested values
  • Graph databases
    • Objects and relationships are modelled and persisted as nodes and edges of a graph

What NoSQL databases are present today?

  • Cassandra
    • Data Model: Columnfamily
    • Query API: Thrift
  • CouchDB
    • Data Model: Document
    • Query API: map/reduce views
  • HBase
    • Data Model: Columnfamily
    • Query API: Thrift, REST
  • MongoDB
    • Data Model: Document
    • Query API: Cursor
  • Neo4j
    • Data Model: Graph
    • Query API: Graph
  • Redis
    • Data Model: Collection
    • Query API: Collection
  • Riak
    • Data Model: Document
    • Query API: Nested hashes
  • Scalaris
    • Data Model: Key/value
    • Query API: get/put
  • Tokyo Cabinet
    • Data Model: Key/value
    • Query API: get/put
  • Voldemort
    • Data Model: Key/value
    • Query API: get/put

.NET APIs

Cassandra

CouchDB

MongoDB

Tokyo Cabinet

Further Reading

A good introduction to the concepts of NoSQL is the paper “NoSQL Databases” from Christof Strauch.
Also read the “Scalable SQL and NoSQL Data Stores” from Rick Cattell.

Link gốc: http://www.codeproject.com/Articles/667371/Working-with-NoSQL-Databases

Implementing Local MemoryCache Invalidation with Redis

We need to use all the tools at our disposal to develop faster and more robust applications. One of the ways we can achieve this is by using caching. Previously, under the System.Web.Caching.Cache namespace, the new  -4.0-System.Runtime.Caching.Memory is more mature and still as powerful as its ancestor. It’s really easy to use and I think that every asp.net programmers have already used it. So, I won’t speak too much about this here.

The default caching strategy is the cache-aside programming pattern: This means that if your data is not present in the cache, your application, and not something else, must reload data into the cache from the original data source.

This works very well for most common use cases, but there is a hidden trade-off: caching means working with stale data. Should I increase the cache duration? Should I keep short TTL value? It’s never easy to answer to these questions, because it simply depends on your context: number of clients, user load, database activity…

Another common problem is how to update/remove instantly something from the cache on all your cache clients, such as this typical request from Product Owner ” I want a parameter to be instantly updated on all our XXX servers, but do not fear, I will only change it a few times per year”

I will investigate a not so new concept in this post: setup local cache invalidation using Redis.

What is Local Cache invalidation ?

Cache invalidation is a process whereby entries in a cache are removed/deleted.
This concept is not specific to .net but to all technologies.
The idea here is to send invalidation message to each server and do not wait –as usual- item expiration.

Why Redis ?

Redis is an open-source, networked, in-memory, key-value data store with optional durability. (Note familiar? Read the great documentation here). It’s now the most popular NoSql database, and many internet  leaders are using it (Instagram, Stackoverflow, Twitter, GitHub, Tumblr, Pinterest, …). For the joke, nothing will be stored in Redis, and we will only use the PubSub feature.

And yes, since I discovered Redis, I’m a big fan :-)

Implementation

Requires : .net, Redis & Booksleeve (high perf C# redis client).

The concept is quite handy :  Redis events (pub/sub) give us  an easy way to broadcast something to all nodes (lossely coupled), so they can drop their local copy – meaning: next time it is needed we’ll pick up the new copy from the data tier.

So, The redis pub/sub events are used to invalidate the cache for a given key from one node (the one that knows the state has changed) immediately to all nodes. A node could be a front web site, a back app or any redis client.

Local cache invalidation is done with ChangeMonitor class : “Provides a base class for a derived custom type that monitors changes in the state of the data which a cache item depends on”. It’s the base class of FileChangeMonitor and SqlChangeMonitor.

redisinvalidation

A few details:

  • A single connection is opened to redis, as BookSleeve is a thread-safe multiplexer
  • As there is a single connection to redis and possibly many monitors, I use a topic-based observer pattern to manage notification inside the application
  • I send only invalidation message following this format :  channel = “invalidate”, and message = “keytoremove”.

Here is a possible implementation in an asp.net application

  • Redis Configuration Configuration
// somewhere in your global.asax (for an asp.net application)
// be sure a redis instance is running on localhost
// check /tools/redis-server.exe
CacheInvalidation.UseRedis(new RedisConnectionInfo() {
     Host = "localhost"
});
  • How to create a redis changemonitor
// somewhere in your code
var cacheItem = new CacheItem("onekey", "onevalue");
var policy = new CacheItemPolicy();
policy.ChangeMonitors.Add(CacheInvalidation.CreateChangeMonitor(cacheItem));
// just to create not expirable item
policy.AbsoluteExpiration = DateTime.Now.AddYears(1);
MemoryCache.Default.Add(cacheItem, policy);

Various Benefits

  • Not limited to .net, and any redis client could invalidate cache item on subscribers
  • Multi-Memory Cache support and not only the default
  • Supports out-of-process invalidation
  • Fully compatible with existing applications, as the invalidation is done by a separate piece of code, in another thread in asp.net
  • Allow you to invalidate a list of items
  • Redis connection unavailable ?
    • Temporary failure : Not a problem as the bus will try to reconnect automatically
    • Redis server not available : No-op, and your application will run without any problems

Knowing how to invalidate, will not resolve all your problems :-)

“There are only two hard problems in Computer Science: cache invalidation and naming things.”, Phil Karlton.

Further considerations …

What about combining local cache invalidation with an out-of-Process caching system?

There is where thing gets interesting: Keep the best of the two worlds and implement a 2-tier cache – i.e. using local memory and the out-of-process cache (distributed).

This is well explained by Marc Gravell, creator of BookSleeve,

“Going off to another system is never faster than querying local memory (as long as it is keyed); (@Stackoverflow) we use both!”

Finally, this also a funny way to introduce Redis in a non-critical scenario. So do not hesitate.

The GitHub Repo is available here.

Link gốc: http://www.codeproject.com/Articles/704387/Implementing-Local-MemoryCache-Invalidation-with-R

Running Redis as a Windows Service

Introduction

In this article I will show how to run the Window version of Redis Server, or other executables, as a Windows service.

Here at CodeProject, we use Redis as a distributed cache. We store massive amounts of information such as Articles, Forum Messages and retrieve these items from the cache rather than the database.

Redis alllows us to store and retrieve full documents, rather than querying SQL for the various pieces and composing and formatting the document on each request. This is possible on our site because most of the information is read a lot more than it is written, and some information, such as number of views, can be a little stale.

While in production we run Redis on a Linux server, in out development environment we are running the Microsoft port of Redis on a Window 7 desktop. The problem is that Redis is not designed to be run as a Windows Service. This means that someone had to logon and run the Redis executable. This was fine when Chris was the only one logging into the server, but last week, I had to connect to install a copy of our Search Server for development purposes. Needless to say, this logged Chris off, and killed the Redis Server. I restarted it under my session.

Back on my machine, I am fixing a subtle caching bug, and when I start testing, the code is acting like there is a connection failure to the Redis Server. As the code I am changing is related to the detection of problems with the connection to Redis, I spin my wheels for half an hour or so before I realize that Chris has remoted into the server, killing my session and the Redis Server.

Both Chris and I tried a number of recommended methods for running Redis as a Windows Service, without any success. Having written several Windows Services to support various CodeProject processes, I decided to write a utility which would allow us to install and run an exe as a Windows Service. This utility is called Exe2Srvc.

Using Exe2Srvc

Exe2Srv is a program that can be run as either a console application or installed as a Windows Service. This application reads the path to an executable, and the command line arguments from it .config file and then starts a the executable in a new Process.

The simplest way to use the executable is to

  1. copy the files in the binfiles download, or from bin/Release from the compiled source, into the directory containing the executable you wish to run as a Service.
  2. Edit the “Cmd” and “CmdArgs” values in the Exe2Srvc.exe.config to contain the full path to the executable, and the command line arguments required.
  3. Run the Install.bat file from a ‘Run as Administrator’ command shell.
  4. Use the Service Manager to:
    • set the start mode to Automatic
    • set the Recovery options. I usually set them to “Restart Service”.
    • start the Service.

For my tests, I downloaded Redis from Nuget, and copied the files from the /packages/Redis-64.2.6.12.1/tools under the solution directory to C:/Redis. I then copied the files fromExe2Srvc\bin\Release to the same directory.

The Exe2Srvc.exe.config file contains:

<?xml version="1.0" encoding="utf-8" />
<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.1" />
    </startup>
    <appSettings>
        <!-- set Cmd to the executable file name. -->
        <!-- set CmdArg to any command arguments required. -->
        <add key="Cmd"      value="c:\Redis\redis-server.exe"/>
        <add key="CmdArgs"  value="c:\Redis\redis.conf --port 6379"/> 
    </appSettings>
</configuration>

If the path to the executable includes spaces, then the path needs to be in quotes. This can be accomplished by enclosing the double-quoted string in single-quotes as shown below.

<?xml version="1.0" encoding="utf-8" />
<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.1" />
    </startup>
    <appSettings>
        <!-- set Cmd to the executable file name. -->
        <!-- set CmdArg to any command arguments required. -->
        <add key="Cmd"      value='"c:Program Files\Redis\redis-server.exe"'/>
        <add key="CmdArgs"  value="c:\Redis\redis.conf --port 6379"/> 
    </appSettings>
</configuration>

The Install.bat file, which must be run as an Administrator, uses the SC command to install the Service.

sc create Redis binpath= "c:\Redis\Exe2Srvc.exe"

Then I use the Services Manager to configure the Service and start it.

Service Manager

Double Click on the Redis service to open the properties editor and select the Recovery tab. Set the options to restart the service on errors as shown.

 Service Manager

Select the General tab.

  • Set the Startup Type to Automatic,
  • click the Start button
  • and click the OK button.
Service Manager

Redis is now installed and running as a Windows Service. Use the redis-cli.exe to test the connection.

How Exe2Srvc works

Creating a console application that can be run as a Windows Service used a technique shown to me by Steve Smith and based on an article Run Windows Service as a console program by Einar Egilsson.

Basically you create a console application, and change the Program class to derive from ServiceBase. In the Main, the Environment.UserInteractive property is used to determine whether the program is being run from the command line, or run as a Service.

The required command and commandline parameters are read from the LoadConfiguration method.

/// <summary>
/// Loads the configuration parameters from the application config file
/// </summary>
private void LoadConfiguration
{
    // Load the executable filename and command arguements from config file.
    _cmd     = ConfigurationManager.AppSettings["Cmd"];
    _cmdArgs = ConfigurationManager.AppSettings["CmdArgs"];
 
    if (string.IsNullOrWhiteSpace(_cmd))
        throw new Exception("The appsetting 'Cmd' was not defined in the config file.");
}

This is called from the OnStart method which is called from Main when started as a console application, or by theService infrastructure when run as a Service. OnStart runs the executable in a new Process as shown below.

/// <summary>
/// When implemented in a derived class, executes when a Start command is sent to the
/// service by the Service Control Manager (SCM) or when the operating system starts
/// (for a service that starts automatically).
/// Specifies actions to take when the service starts.
/// </summary>
/// <param name="args";>
/// Data passed by the start command.
/// </param>
protected override void OnStart(string[] args)
{
    if (Environment.UserInteractive)
    {
        string message = String.Format"Starting {0} at {1}.", _serviceName, DateTime.Now);
         Console.WriteLine(message);
    }
 
    // loading the configuration file info here allows the service to be stopped,
    // the configuration modified, and the service restarted.
    LoadConfiguration();
 
     // Start the executable.
      ProcessStartInfo procInfo = new ProcessStartInfo(_cmd);
      procInfo.UseShellExecute = false;
 
       if (!string.IsNullOrWhiteSpace(_cmdArgs))
           procInfo.Arguments = _cmdArgs;
 
       _process = Process.Start(procInfo);
 }
 

When the Service is stopped, the OnStop method is called. This kills the Process, waits for it to terminate, and disposes of the Process. Make sure you wait for the Process to terminate, failure to do so will result in improperly stopped Services that are difficult to remove and fix.

/// <summary>
/// When implemented in a derived class, executes when a Stop command is sent to the service
/// by the Service Control Manager (SCM). Specifies actions to take when a service stops running.
/// </summary>
/// <remarks>Stops the background tasks.</remarks>
protected override void OnStop()
{
    if (Environment.UserInteractive)
    {
        string message = String.Format("Stopping {0} at {1}.", _serviceName, DateTime.Now);
         Console.WriteLine(message);
     }
           
     // Kill the process
     if (_process != null)
     {
         _process.Kill();
         _process.WaitForExit();
         _process.Dispose();
         _process = null;
      }
}

Points of Interest

I’ve attempted to make this as simple and flexible as possible, but it only meets my original requirement of being able to run the Windows version of Redis as a Windows Server. If you have additional requirement, feel free to modify the code to match your needs.

Link gốc:  http://www.codeproject.com/Articles/715967/Running-Redis-as-a-Windows-Service

Distributed Caching using Redis Server with .NET/C# Client

Introduction

In this article, I would like to describe my experience with installing and configuring Redis server in most compact way. Also, I would like to do a brief overview of usage Redis hashes and lists in .NET/C# client.

In this article:

Background

Redis is one of the fastest and most feature-rich in-memory key value data stores.

Disadvantages

  • No local data buffering (like synchronized local data buffering on the Azure caching)
  • No full clusterization support yet (expected in the end of this year)

Advantages

  • Easy configuration
  • Simple usage
  • High performance
  • Support of different data types (like hashes, lists, sets, sorted sets)
  • ASP.NET session integration
  • Web UI for viewing content of the cache

In this simple demo, I’m going to demonstrate how to install and configure Redis on the server and use it from the C# code.

Redis Installation

Download binaries from https://github.com/dmajkic/redis/downloads (win32 win64 direct link) unpack archive to the application directory (e.g. C:\Program Files\Redis)

Download Redis service compiled from https://github.com/kcherenkov/redis-windows-service/downloads, then copy to the program folder (e.g. C:\Program Files\Redis. If config file is missing, download it and copy to the application directory as well. Example of the valid Redis config file is at https://raw.github.com/antirez/redis/2.6/redis.conf.

The complete set of files for Redis application is also available in zip file (x64).

When you have the full set of the application files (like it is shown on the image below),

redis application folder conten

navigate to the application directory and run the following command:

sc create %name% binpath= "\"%binpath%\" %configpath%" start= "auto" DisplayName= "Redis"

Where:

  • %name% — name of service instance, example: redis-instance;
  • %binpath% — path to this project EXE file, example: C:\Program Files\Redis\RedisService_1.1.exe;
  • %configpath% — path to redis configuration file, example: C:\Program Files\Redis\redis.conf;

Example:

sc create Redis start= auto DisplayName= Redis binpath= "\"C:\Program Files\Redis\RedisService_1.1.exe\
" \"C:\Program Files\Redis\redis.conf\""

It should look like this:

Installing redis as windows service

Make sure that you have enough privileges to start the service. After installation, check that the service was created successfully and is running now:

redis running as a windows service

Alternatively, you can use installer, created by someone (I’ve not tried): https://github.com/rgl/redis/downloads.

Redis Server Protection: Password, IP Filtering

The primary way to protect Redis server is to set IP filtering using Windows firewall or properties of the active network connection. Additional protection can be set using redis password. It needs to update the Redis config file (redis.conf) in the following way:

First, find the line:

# requirepass foobared

Remove the # symbol in the beginning and replace foobared with new password:

requirepass foobared

Then restart Redis Windows service!!!

When instantiating the client, use constructor with a password:

RedisClient client = new RedisClient(serverHost, port, redisPassword);

Redis Server Replication (master – slave configuration)

This technique allows creation copy of the server data into the synchronized copy, this means that each time when master is modified, slave server gets notification and is automatically synchronized. Mostly replication is used for read (but not write) scalability or data redundancy and for the server failover. Setup two instances of Redis (two services on the same or different servers), then configure one of them as slave. To make Redis server instance to be slave of another server, change the config file in this way:

Find the line below:

# slaveof <masterip> <masterport>

replace with:

slaveof 192.168.1.1 6379

(specify real IP of the master server, and port in case you customized it). If master is configured to require password (authentication), change redis.conf as it is shown below, find line:

# masterauth <master-password>

remove the # symbol in the beginning and replace <master-password> with master password, to be like that:

masterauth mastpassword

Now this Redis instance can be used as a readonly synchronized copy of the master server.

Using Redis Cache from the C# Code

To use Redis in C# run the Manage NuGet packages addon, find ServiceStack.Redis pack, and install it.

Sample of using Set/Get methods directly from the instantiated client:

string host = "localhost";
string elementKey = "testKeyRedis";

using (RedisClient redisClient = new RedisClient(host))
{
      if (redisClient.Get<string>(elementKey) == null)
      {
           // adding delay to see the difference
           Thread.Sleep(5000); 
           // save value in cache
           redisClient.Set(elementKey, "some cached value");
      }
      // get value from the cache by key
      message = "Item value is: " + redisClient.Get<string>("some cached value");
 }

Typed entity sets are more interesting and practical, because they operate with exact types of objects. In the code sample below, there are two classes defined Phone, and Person – owner of the phone. Each phone instance has a reference to the owner. This code demonstrates how we can add, remove or find items in the cache by criteria:

public class Phone
{
   public int Id { get; set; }
   public string Model { get; set; }
   public string Manufacturer { get; set; }
   public Person Owner { get; set; }
}

public class Person
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string Surname { get; set; }
    public int Age { get; set; }
    public string Profession { get; set; }
}

using (RedisClient redisClient = new RedisClient(host))
{
     IRedisTypedClient<phone> phones = redisClient.As<phone>();
     Phone phoneFive = phones.GetValue("5");
     if (phoneFive == null)
     {
          // make a small delay
          Thread.Sleep(5000);
          // creating a new Phone entry
          phoneFive = new Phone
          {
               Id = 5,
               Manufacturer = "Motorolla",
               Model = "xxxxx",
               Owner = new Person
               {
                    Id = 1,
                    Age = 90,
                    Name = "OldOne",
                    Profession = "sportsmen",
                    Surname = "OldManSurname"
               }
          };
          // adding Entry to the typed entity set
          phones.SetEntry(phoneFive.Id.ToString(), phoneFive);
     }
     message = "Phone model is " + phoneFive.Manufacturer;
     message += "Phone Owner Name is: " + phoneFive.Owner.Name;
}

In the example above, we instantiate the typed client IRedisTypedClient, which works with specific type of cached objects: Phone type.

ASP.NET Session State with Redis

To configure ASP.NET session state with redis provider, add a new file to your web project, namedRedisSessionStateProvider.cs, copy code from https://github.com/chadman/redis-service-provider/raw/master/RedisProvider/SessionProvider/RedisSessionProvider.cs, then add or change the following section in the configuration file (sessionState tag has to be inside system.web tag), or you can download attached sources and copy code.

<sessionstate timeout="1" mode="Custom" 
customprovider="RedisSessionStateProvider" cookieless="false">
      <providers>
        <add name="RedisSessionStateProvider" writeexceptionstoeventlog="false" 
        type="RedisProvider.SessionProvider.CustomServiceProvider" 
        server="localhost" port="6379" password="pasword">
      </add> </providers>
</sessionstate>

NOTE, that password is optional, based on the server authentication. It must be replaced with real value, or removed, if Redis server doesn’t require authentication. server attribute and port also have to be replaced according to concrete values (default port is 6379). Then in the project, you can use the session state:

// in the Global.asax
public class MvcApplication1 : System.Web.HttpApplication
{
    protected void Application_Start()
    {
        //....
    }

    protected void Session_Start()
    {
        Session["testRedisSession"] = "Message from the redis ression";
    }
}

In the Home controller:

public class HomeController : Controller
{
    public ActionResult Index()
    {
       //...
       ViewBag.Message = Session["testRedisSession"];
       return View();
    }
//...
}

Result:

redis aspnet session state

ASP.NET output cache provider with redis can be configured in the similar way.

Redis Sets and Lists

The major note is that Redis lists implement IList<T> while Redis sets implement ICollection<T>. Let’s see how we can use them.

Lists are mostly used when it needs to separate different categories of objects of the same type. For example, we have “most selling phones” and “old collection” two lists of phones:

string host = "localhost";
using (var redisClient = new RedisClient(host))
{
    //Create a 'strongly-typed' API that makes all Redis Value operations to apply against Phones
    IRedisTypedClient<phone> redis = redisClient.As<phone>();

    IRedisList<phone> mostSelling = redis.Lists["urn:phones:mostselling"];
    IRedisList<phone> oldCollection = redis.Lists["urn:phones:oldcollection"];

    Person phonesOwner = new Person
        {
            Id = 7,
            Age = 90,
            Name = "OldOne",
            Profession = "sportsmen",
            Surname = "OldManSurname"
        };
                
    // adding new items to the list
    mostSelling.Add(new Phone
            {
                Id = 5,
                Manufacturer = "Sony",
                Model = "768564564566",
                Owner = phonesOwner
            });

    mostSelling.Add(new Phone
            {
                Id = 8,
                Manufacturer = "Motorolla",
                Model = "324557546754",
                Owner = phonesOwner
            });

    var upgradedPhone  = new Phone
    {
        Id = 3,
        Manufacturer = "LG",
        Model = "634563456",
        Owner = phonesOwner
    };

    mostSelling.Add(upgradedPhone);

    // remove item from the list
    oldCollection.Remove(upgradedPhone);

    // find objects in the cache
    IEnumerable<phone> LGPhones = mostSelling.Where(ph => ph.Manufacturer == "LG");

    // find specific
    Phone singleElement = mostSelling.FirstOrDefault(ph => ph.Id == 8);

    //reset sequence and delete all lists
    redis.SetSequence(0);
    redisClient.Remove("urn:phones:mostselling");
    redisClient.Remove("urn:phones:oldcollection");
}

Redis sets are useful when it needs to store associated sets of data and gather statistical information, for example answer -> queustion, votes for an answer or question. Let’s say that we have questions and answers, it needs to store them in the cache for better performance. Using Redis, we can do it this way:

/// <summary>
/// Gets or sets the Redis Manager. The built-in IoC used with ServiceStack autowires this property.
/// </summary>
IRedisClientsManager RedisManager { get; set; }
/// <summary>
/// Delete question by performing compensating actions to 
/// StoreQuestion() to keep the datastore in a consistent state
/// </summary>
/// <param name="questionId">
public void DeleteQuestion(long questionId)
{
    using (var redis = RedisManager.GetClient())
    {
        var redisQuestions = redis.As<question>();

        var question = redisQuestions.GetById(questionId);
        if (question == null) return;
                
        //decrement score in tags list
        question.Tags.ForEach(tag => redis.IncrementItemInSortedSet("urn:tags", tag, -1));

        //remove all related answers
        redisQuestions.DeleteRelatedEntities<answer>(questionId);

        //remove this question from user index
        redis.RemoveItemFromSet("urn:user>q:" + question.UserId, questionId.ToString());

        //remove tag => questions index for each tag
        question.Tags.ForEach("urn:tags>q:" + tag.ToLower(), questionId.ToString()));

        redisQuestions.DeleteById(questionId);
    }
}

public void StoreQuestion(Question question)
{
    using (var redis = RedisManager.GetClient())
    {
        var redisQuestions = redis.As<question>();

        if (question.Tags == null) question.Tags = new List<string>();
        if (question.Id == default(long))
        {
            question.Id = redisQuestions.GetNextSequence();
            question.CreatedDate = DateTime.UtcNow;

            //Increment the popularity for each new question tag
            question.Tags.ForEach(tag => redis.IncrementItemInSortedSet("urn:tags", tag, 1));
        }

        redisQuestions.Store(question);
        redisQuestions.AddToRecentsList(question);
        redis.AddItemToSet("urn:user>q:" + question.UserId, question.Id.ToString());

        //Usage of tags - Populate tag => questions index for each tag
        question.Tags.ForEach(tag => redis.AddItemToSet
        ("urn:tags>q:" + tag.ToLower(), question.Id.ToString()));
    }
}

/// <summary>
/// Delete Answer by performing compensating actions to 
/// StoreAnswer() to keep the datastore in a consistent state
/// </summary>
/// <param name="questionId">
/// <param name="answerId">
public void DeleteAnswer(long questionId, long answerId)
{
    using (var redis = RedisManager.GetClient())
    {
        var answer = redis.As<question>().GetRelatedEntities<answer>
        (questionId).FirstOrDefault(x => x.Id == answerId);
        if (answer == null) return;
                
        redis.As<question>().DeleteRelatedEntity<answer>(questionId, answerId);
                
        //remove user => answer index
        redis.RemoveItemFromSet("urn:user>a:" + answer.UserId, answerId.ToString());
    }
}

public void StoreAnswer(Answer answer)
{
    using (var redis = RedisManager.GetClient())
    {
        if (answer.Id == default(long))
        {
            answer.Id = redis.As<answer>().GetNextSequence();
            answer.CreatedDate = DateTime.UtcNow;
        }

        //Store as a 'Related Answer' to the parent Question
        redis.As<question>().StoreRelatedEntities(answer.QuestionId, answer);
        //Populate user => answer index
        redis.AddItemToSet("urn:user>a:" + answer.UserId, answer.Id.ToString());
    }
}

public List<answer> GetAnswersForQuestion(long questionId)
{
    using (var redis = RedisManager.GetClient())
    {
        return redis.As<question>().GetRelatedEntities<answer>(questionId);
    }
}

public void VoteQuestionUp(long userId, long questionId)
{
    //Populate Question => User and User => Question set indexes in a single transaction
    RedisManager.ExecTrans(trans =>
    {
        //Register upvote against question and remove any downvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:q>user+:" + questionId, userId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:q>user-:" + questionId, userId.ToString()));

        //Register upvote against user and remove any downvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:user>q+:" + userId, questionId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:user>q-:" + userId, questionId.ToString()));
    });
}

public void VoteQuestionDown(long userId, long questionId)
{
    //Populate Question => User and User => Question set indexes in a single transaction
    RedisManager.ExecTrans(trans =>
    {
        //Register downvote against question and remove any upvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:q>user-:" + questionId, userId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:q>user+:" + questionId, userId.ToString()));

        //Register downvote against user and remove any upvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet"urn:user>q-:" + userId, questionId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:user>q+:" + userId, questionId.ToString()));
    });
}

public void VoteAnswerUp(long userId, long answerId)
{
    //Populate Question => User and User => Question set indexes in a single transaction
    RedisManager.ExecTrans(trans =>
    {
        //Register upvote against answer and remove any downvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:a>user+:" + answerId, userId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:a>user-:" + answerId, userId.ToString()));

        //Register upvote against user and remove any downvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:user>a+:" + userId, answerId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:user>a-:" + userId, answerId.ToString()));
    });
}

public void VoteAnswerDown(long userId, long answerId)
{
    //Populate Question => User and User => Question set indexes in a single transaction
    RedisManager.ExecTrans(trans =>
    {
        //Register downvote against answer and remove any upvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:a>user-:" + answerId, userId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:a>user+:" + answerId, userId.ToString()));

        //Register downvote against user and remove any upvotes if any
        trans.QueueCommand(redis => 
        redis.AddItemToSet("urn:user>a-:" + userId, answerId.ToString()));
        trans.QueueCommand(redis => 
        redis.RemoveItemFromSet("urn:user>a+:" + userId, answerId.ToString()));
    });
}

public QuestionResult GetQuestion(long questionId)
{
    var question = RedisManager.ExecAs<question>
    (redisQuestions => redisQuestions.GetById(questionId));
    if (question == null) return null;

    var result = ToQuestionResults(new[] { question })[0];
    var answers = GetAnswersForQuestion(questionId);
    var uniqueUserIds = answers.ConvertAll(x => x.UserId).ToHashSet();
    var usersMap = GetUsersByIds(uniqueUserIds).ToDictionary(x => x.Id);

    result.Answers = answers.ConvertAll(answer =>
        new AnswerResult { Answer = answer, User = usersMap[answer.UserId] });

    return result;
}

public List<user> GetUsersByIds(IEnumerable<long> userIds)
{
    return RedisManager.ExecAs<user>(redisUsers => redisUsers.GetByIds(userIds)).ToList();
}

public QuestionStat GetQuestionStats(long questionId)
{
    using (var redis = RedisManager.GetReadOnlyClient())
    {
        var result = new QuestionStat
        {
            VotesUpCount = redis.GetSetCount("urn:q>user+:" +questionId),
            VotesDownCount = redis.GetSetCount("urn:q>user-:" + questionId)
        };
        result.VotesTotal = result.VotesUpCount - result.VotesDownCount;
        return result;
    }
}

public List<tag> GetTagsByPopularity(int skip, int take)
{
    using (var redis = RedisManager.GetReadOnlyClient())
    {
        var tagEntries = redis.GetRangeWithScoresFromSortedSetDesc("urn:tags", skip, take);
        var tags = tagEntries.ConvertAll(kvp => new Tag { Name = kvp.Key, Score = (int)kvp.Value });
        return tags;
    }
}

public SiteStats GetSiteStats()
{
    using (var redis = RedisManager.GetClient())
    {
        return new SiteStats
        {
            QuestionsCount = redis.As<question>().TypeIdsSet.Count,
            AnswersCount = redis.As<answer>().TypeIdsSet.Count,
            TopTags = GetTagsByPopularity(0, 10)
        };
    }
}

Attached Sources Description

List of included packages is in the packages.config,
Funq IoC configuration, registering types and current controller factory – are in the Global.asax (property dependency injection)
Usage of a simple client – in the home controller
Usage of IoC based cache in the Question and Answer controllers, and Global.asax application file. To see how it works, you can run the project, and open in the browser following URL:http://localhost:37447/Question/GetQuestions?tag=test .
You can play with tags, like test3, test11, test2, etc.
Redis Cache configuration – in the web config (<system.web><sessionState> section) and in theRedisSessionStateProvider.cs file.
There are a lot of TODOs in the MVC project, so if you want to improve/continue please update it, and upload.

I would much appreciate if someone could help build MVC application with simple UI, using Redis (with Funq IoC) cache. Funq IoC is already configured, example of the usage is in the Question controller.

NOTE: Samples were partially taken from the “ServiceStack.Examples-master” solution

Conclusion. Optimization Caching in Application with Fast Local Cache

Since Redis doesn’t store data locally (no local replication), it might make sence to optimize performance by storing some light or user – dependent objects in the local cache (to skip serialization to string and client – server data transfering). For example, in the web application, it is better to use ‘System.Runtime.Caching.ObjectCache‘ for light objects, which are user dependent and used frequently by the application. Otherwise, when object has common usage, large size it must be saved in the distributed Redis cache. Example of the user dependent objects – profile information, personalization information. Common objects – localization data, information shared between different users, etc.

LINK gốc:  http://www.codeproject.com/Articles/636730/Distributed-Caching-using-Redis

USB Reset Attribute

Introduction

What do you do when your Removal Device or USB gets affected by a virus and files are hidden? Most of the Antivirus scan the device and delete the virus only. So we can’t see the files in our USB device which is hidden by the virus. So here, the “Ultimate USB Reset Attribute” comes with a powerful option.

Idea Behind the Code

I just embed my code with the existing one which is already available in here (CodeProject). So my first sincere thanks to CodeProject, without this I could not do it.

Using the Code

Using this code, the number of USB or removal devices is listed in a dropdown which is currently connected to a computer.

if(!Directory .Exists (currDir ))
            return;
            //set the progress bar max value
            pbar.Maximum += Directory.GetFileSystemEntries(currDir).Count();
            //iterate over the files
            foreach (string ff in Directory.GetFiles(currDir))
            {   //super cast to the parent class FileSystemInfo
                ResetAttribute((new FileInfo(ff)) as FileSystemInfo );
             }
          
            foreach (string dr in Directory.GetDirectories(currDir))
            {
                if (!Directory.Exists(dr))
                    return;
                //reset the directory attribute
               ResetAttribute((new DirectoryInfo(dr)) as FileSystemInfo);
                      ScanFiles(dr); //scan the directory  
//Changing the file attributes by using this code.
try
            { //bitwise operator on the file attribute
                fsi.Attributes &= ~FileAttributes.Hidden;
                fsi.Attributes &= ~FileAttributes.System;

                fsi.Attributes &= ~FileAttributes.ReadOnly;
            pbar.Value += 1;
            }
            catch (Exception ex)
            {
                return;
            } 
//Whenever the main form is loaded, it checks and gets drive details by using this code.
 var drv = from dr in DriveInfo.GetDrives()
                     where dr.IsReady == true && dr.DriveType == DriveType.Removable
                     select dr;
            foreach (DriveInfo dinfo in drv)
            {
                cmbUSB.Items.Add(dinfo.Name);
                cmbUSB.SelectedIndex = 0;
              } 
 
  DriveInfo dr = new DriveInfo(cmbUSB.SelectedItem.ToString());
    lblusb.Text = dr.VolumeLabel + " (" + dr.Name + ")";

Command Line Reseter

Just enter the Drive Letter what you want to reset. Using this code.

set /p letter=
attrib -s -h -a /s /d %letter%:\*.*

Points of Interest

Hope this version helps you. This is the general idea, which I got in my dreams!!!

Screenshots

 

Link: http://www.codeproject.com/Tips/755487/USB-Reset-Attribute