Leveraging In-Memory Key Value Stores for Large-Scale Operations

Memcache, Redis, and most other in-memory key-value systems have traditionally been used to offload (scale) queries against backend databases. Facebook made this architecture famous, showing that it is possible to have thousands of Web servers requests satisfied in sub-second time by standing up in-memory caches in front of databases. At LinkedIn, we have taken usage of in-memory caches in a completely opposite direction—we leverage them to solve operational questions:

• Where does the httpd process run?
• What versions of the openssh package is installed in datacenter X?
• Who has a network connection to machine Y?
• What machines have experienced hardware failure?

By standing up Redis Caches on each of our CFEngine policy servers, every client populates Redis caches on every execution of CFEngine. We have built a Python library at LinkedIn where we leverage our "Range" lookup system to perform distributed queries against Redis on 60x policy servers in parallel. This approach allows us to answer any question about our infrastructure and have results delivered in under five seconds from tens or hundreds of thousands of machines. It allows our security team to find machines that could have been exploited, allows our SRE team to understand where services have been deployed, and helps SysOps build our inventory database system and modify our CMDB in real time.

Mike Svoboda currently works in System Operations at LinkedIn and is charged with administrating all production automation. LinkedIn relies on CFEngine to tie major parts of infrastructure together, which has allowed LinkedIn the flexibility to scale horizontally indefinitely.

Diego Zamboni is a computer scientist, consultant, author, programmer, sysadmin, and overall geek who works as a senior security advisor at CFEngine. He has more than 20 years of experience in system administration and security, and has worked in both the applied and theoretical sides of the computer science field. Zamboni is the author of the book Learning CFEngine 3, published by O’Reilly Media.