"拥抱变化，如今也走上了.net/java 通吃的时代，下面就讲讲如何让.net/java 都能正常访问分片的 redis 吧。有几个关键点：一致性环哈希、哈希算法、序列化、反序列化后两个都比较直接，只要选择一种跨语言的序列化方式就行了，如：json, protobuf, ace 等，本文全略了 ...."

someone1
AI专业者开发社区 1 号成员
AI开发者社区 • 0 回帖 • 0 浏览 • 5 个月前

C# servicestack.redis 互通 java jedis

拥抱变化，如今也走上了.net/java 通吃的时代，下面就讲讲如何让.net/java 都能正常访问分片的 redis 吧。

有几个关键点：一致性环哈希、哈希算法、序列化、反序列化

后两个都比较直接，只要选择一种跨语言的序列化方式就行了，如：json, protobuf, ace 等，本文全略了

本文是基于 jedis 的一致性环哈希来修改的，.net 选的是 servicestack.redis 组件来修改

无奈两个组件都有各自的一致性环哈希算法，不兼容，那就选一个作为标准，修改另一个咯。本文选择 jedis 的一致性环哈希作为标准，进而修改.net 来适应 jedis

jedis 的逻辑是给每个 redis 节点构造 160 个虚拟节点，放入一颗二叉树中（key/value：key 是一个 long 值，根据哈希算法算出来的一个 long、value 是节点 id，是个 string）。

OK，逻辑清楚了，那就简单了，给 c# 端写个一模一样的一致性环哈希算法。

public class Sharded
    {
        private object nodes_lock = new object();
        private RedBlackTreeMap<long, string> nodes = new RedBlackTreeMap<long, string>();
        private IHash hashAlgo = new MD5_LongSUM_Multiply_Hash();
    </span><span style="color: rgba(0, 0, 255, 1)">public</span> <span style="color: rgba(0, 0, 255, 1)">void</span> AddTarget(<span style="color: rgba(0, 0, 255, 1)">int</span> index, <span style="color: rgba(0, 0, 255, 1)">string</span><span style="color: rgba(0, 0, 0, 1)"> shard)
    {
        </span><span style="color: rgba(0, 0, 255, 1)">lock</span><span style="color: rgba(0, 0, 0, 1)"> (nodes_lock)
        {
            </span><span style="color: rgba(0, 0, 255, 1)">for</span> (<span style="color: rgba(0, 0, 255, 1)">int</span> n = <span style="color: rgba(128, 0, 128, 1)">0</span>; n &lt; <span style="color: rgba(128, 0, 128, 1)">160</span>; ++<span style="color: rgba(0, 0, 0, 1)">n)
            {
                </span><span style="color: rgba(0, 0, 255, 1)">var</span> hashKey = <span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)">SHARD-</span><span style="color: rgba(128, 0, 0, 1)">"</span> + index + <span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)">-NODE-</span><span style="color: rgba(128, 0, 0, 1)">"</span> +<span style="color: rgba(0, 0, 0, 1)"> n;

                </span><span style="color: rgba(0, 0, 255, 1)">long</span> hashValue = <span style="color: rgba(0, 0, 255, 1)">this</span><span style="color: rgba(0, 0, 0, 1)">.hashAlgo.Hash(hashKey);

                nodes.SetOrAddValue(hashValue, shard);
            }
        }
    }


    </span><span style="color: rgba(0, 0, 255, 1)">public</span> <span style="color: rgba(0, 0, 255, 1)">string</span> GetShardInfo(<span style="color: rgba(0, 0, 255, 1)">string</span><span style="color: rgba(0, 0, 0, 1)"> key)
    {
        </span><span style="color: rgba(0, 0, 255, 1)">long</span> searchHashKey = <span style="color: rgba(0, 0, 255, 1)">this</span><span style="color: rgba(0, 0, 0, 1)">.hashAlgo.Hash(key);

        </span><span style="color: rgba(0, 0, 255, 1)">long</span><span style="color: rgba(0, 0, 0, 1)"> nearestKey;
        </span><span style="color: rgba(0, 0, 255, 1)">string</span><span style="color: rgba(0, 0, 0, 1)"> shard;

        </span><span style="color: rgba(0, 0, 255, 1)">lock</span><span style="color: rgba(0, 0, 0, 1)"> (nodes_lock)
        {
            </span><span style="color: rgba(0, 0, 255, 1)">if</span> (<span style="color: rgba(0, 0, 255, 1)">this</span>.nodes.NearestGreater(searchHashKey, <span style="color: rgba(0, 0, 255, 1)">out</span><span style="color: rgba(0, 0, 0, 1)"> nearestKey))
            {
                shard </span>= <span style="color: rgba(0, 0, 255, 1)">this</span><span style="color: rgba(0, 0, 0, 1)">.nodes.GetValue(nearestKey);
                </span><span style="color: rgba(0, 0, 255, 1)">return</span><span style="color: rgba(0, 0, 0, 1)"> shard;
            }

            </span><span style="color: rgba(0, 0, 255, 1)">if</span> (<span style="color: rgba(0, 0, 255, 1)">this</span>.nodes.Least(<span style="color: rgba(0, 0, 255, 1)">out</span> searchHashKey, <span style="color: rgba(0, 0, 255, 1)">out</span><span style="color: rgba(0, 0, 0, 1)"> shard))
                </span><span style="color: rgba(0, 0, 255, 1)">return</span><span style="color: rgba(0, 0, 0, 1)"> shard;
        }

        </span><span style="color: rgba(0, 0, 255, 1)">throw</span> <span style="color: rgba(0, 0, 255, 1)">new</span> Exception(<span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)">GetShardInfo exception</span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(0, 0, 0, 1)">);
    }
}</span></pre>

其中 RedBlackTreeMap 这个是 TreeLib 中的组件，需要在 nuget 上引用。

MD5_LongSUM_Multiply_Hash，这是个 MD5 算法，输入为 string，输出为 long。
此处由于考虑到输出不是 string，因此自己又改了改，让他输出 long

public class MD5_LongSUM_Multiply_Hash : IHash
    {
        public long Hash(string key)
        {
            var md5= Md5Hash(key);
        </span><span style="color: rgba(0, 0, 255, 1)">if</span> (<span style="color: rgba(0, 0, 255, 1)">string</span><span style="color: rgba(0, 0, 0, 1)">.IsNullOrEmpty(md5))
            Log.GetLog().Info(</span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)">Hash, md5 is null or empty</span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(0, 0, 0, 1)">);

        </span><span style="color: rgba(0, 0, 255, 1)">var</span> convertedKeyBytes =<span style="color: rgba(0, 0, 0, 1)"> Encoding.UTF8.GetBytes(md5);
        
        </span><span style="color: rgba(0, 0, 255, 1)">long</span> value = <span style="color: rgba(128, 0, 128, 1)">1</span><span style="color: rgba(0, 0, 0, 1)">;

        </span><span style="color: rgba(0, 0, 255, 1)">foreach</span>(<span style="color: rgba(0, 0, 255, 1)">var</span> b <span style="color: rgba(0, 0, 255, 1)">in</span><span style="color: rgba(0, 0, 0, 1)"> convertedKeyBytes)
            value </span>*= b*-<span style="color: rgba(128, 0, 128, 1)">1</span><span style="color: rgba(0, 0, 0, 1)">;

        </span><span style="color: rgba(0, 0, 255, 1)">return</span><span style="color: rgba(0, 0, 0, 1)"> value;
    }

    </span><span style="color: rgba(0, 0, 255, 1)">private</span> <span style="color: rgba(0, 0, 255, 1)">string</span> Md5Hash(<span style="color: rgba(0, 0, 255, 1)">string</span><span style="color: rgba(0, 0, 0, 1)"> input)
    {
        MD5CryptoServiceProvider md5Hasher </span>= <span style="color: rgba(0, 0, 255, 1)">new</span><span style="color: rgba(0, 0, 0, 1)"> MD5CryptoServiceProvider();

        </span><span style="color: rgba(0, 0, 255, 1)">if</span> (<span style="color: rgba(0, 0, 255, 1)">string</span><span style="color: rgba(0, 0, 0, 1)">.IsNullOrEmpty(input))
            Log.GetLog().Info(</span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)">Md5Hash, input is null or empty</span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(0, 0, 0, 1)">);

        </span><span style="color: rgba(0, 0, 255, 1)">byte</span>[] data =<span style="color: rgba(0, 0, 0, 1)"> md5Hasher.ComputeHash(Encoding.Default.GetBytes(input));
        StringBuilder sBuilder </span>= <span style="color: rgba(0, 0, 255, 1)">new</span><span style="color: rgba(0, 0, 0, 1)"> StringBuilder();
        </span><span style="color: rgba(0, 0, 255, 1)">for</span> (<span style="color: rgba(0, 0, 255, 1)">int</span> i = <span style="color: rgba(128, 0, 128, 1)">0</span>; i &lt; data.Length; i++<span style="color: rgba(0, 0, 0, 1)">)
        {
            sBuilder.Append(data[i].ToString(</span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)">x2</span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(0, 0, 0, 1)">));
        }
        </span><span style="color: rgba(0, 0, 255, 1)">return</span><span style="color: rgba(0, 0, 0, 1)"> sBuilder.ToString();
    }
}</span></pre>

剩下的就是 java 端的这个输入 string，输出 long 的算法，需要和.net 的输入输出一致了。

那就也写一个哈希算法，让他输入 string，输出 long，和.net 的一致，这里只要 java/.net 用同一种 md5 算法，后续的 md5 变成 long 就很容易了。

import org.springframework.security.authentication.encoding.MessageDigestPasswordEncoder;
import redis.clients.util.Hashing;
import redis.clients.util.SafeEncoder;
import java.io.UnsupportedEncodingException;
/**


Created by z on 2017/4/12.
*/
public class MD5_SUM_Hash implements Hashing {
MessageDigestPasswordEncoder encoder=new MessageDigestPasswordEncoder("MD5");
public long hash(String key) {
return this.hash(SafeEncoder.encode(key));
}
public long hash(byte[] bytes) {
 String converted_str</span>= <span style="color: rgba(0, 0, 255, 1)">null</span><span style="color: rgba(0, 0, 0, 1)">;
 </span><span style="color: rgba(0, 0, 255, 1)">try</span><span style="color: rgba(0, 0, 0, 1)"> {
     converted_str </span>= <span style="color: rgba(0, 0, 255, 1)">new</span> String(bytes, "UTF8"<span style="color: rgba(0, 0, 0, 1)">);
 } </span><span style="color: rgba(0, 0, 255, 1)">catch</span><span style="color: rgba(0, 0, 0, 1)"> (UnsupportedEncodingException e) {
     e.printStackTrace();
 }

 String result</span>=encoder.encodePassword(converted_str, <span style="color: rgba(0, 0, 255, 1)">null</span><span style="color: rgba(0, 0, 0, 1)">);

 </span><span style="color: rgba(0, 0, 255, 1)">try</span><span style="color: rgba(0, 0, 0, 1)"> {
     bytes</span>=result.getBytes("UTF8"<span style="color: rgba(0, 0, 0, 1)">);
 } </span><span style="color: rgba(0, 0, 255, 1)">catch</span><span style="color: rgba(0, 0, 0, 1)"> (UnsupportedEncodingException e) {
     e.printStackTrace();
 }

 </span><span style="color: rgba(0, 0, 255, 1)">long</span> value = 1<span style="color: rgba(0, 0, 0, 1)">;
 </span><span style="color: rgba(0, 0, 255, 1)">for</span>(<span style="color: rgba(0, 0, 255, 1)">byte</span><span style="color: rgba(0, 0, 0, 1)"> b : bytes)
     value </span>*= b*-1<span style="color: rgba(0, 0, 0, 1)">;
 </span><span style="color: rgba(0, 0, 255, 1)">return</span><span style="color: rgba(0, 0, 0, 1)"> value;

}
}

<dependency>
            <groupId>org.springframework.security</groupId>
            <artifactId>spring-security-core</artifactId>
        </dependency>

OK，核心的就这些了。

C# servicestack.redis 互通 java jedis

相关帖子

随便看看

C# servicestack.redis 互通 java jedis