Image may be NSFW.
Clik here to view.
I went to RedisConf 2017 Wed. and Thu. at the Marriott Marquis in SF hosted by RedisLabs.
About 2,000 people showed up – initially surprising to me, but then Redis is more than a key-value store.
Kudos to RedisLabs for a great conference: excellent venue, very well-organized, good food but intermittent Wifi.
Special credit for the combination of excellent, very technical talks with “lateral” keynotes that were thought-provoking even when not Redis-specific.
Conference slides are here.
Executive Summary
- 2017 is the year of three modern “at scale” tools: Redis, Spark and Grafana.
- Redis is much more than a sub-millisecond key-value store. It has several killer features, including zsets, geo calculations and pub/sub. Redis can tier storage to SSD, supporting multi-TB datasets that are larger than RAM.
- Redis users are already using replication and clustering features. Coming soon is multi-master with vector clocks, enabling multi-region masters, the Holy Grail of data persistence.
- Caches should be managed (documented, monitored and administered) by assigned operations staff, and in more complex scenarios, by DBAs.
Wednesday Keynotes
Kelsey Hightower, Staff Developer Advocate, Google
– Kelsey built redis kubernetes cluster in live demo and did failures
– used “Siri” voice commands to do some of the deployment
Sam Ramji, VP Product Management, Google
– very interesting high-level comments
– Dr. Eric Brewer is a VP Eng.
– “Spanner defeats CAP” because:
1) Google’s network is reliable (no partitions expected)
2) atomic clocks in each DC, less than 200 microsecond drift
– Spanner has SQL UI now, so anybody can access data.
Image may be NSFW.
Clik here to view.
Spanner High-Level Architecture Diagram
Wednesday Talks
Case Study: Redis Cluster at Flickr (Yahoo!)
Sean Perkins, Sr. Operations Engineer (18-year veteran)
– traditional large-scale IT environment (still pets not cattle with long-lived masters and names/IP addresses)
– had to do a cleanup and doc phase, dedicated vs. general cache instances
– recommends rcm esp. since hard to tell if redis cluster nodes up-to-date
– managed with yinst, chef, jenkins/screwdriver
– not full-on HA, but good enough for easy upgrades
– no redis cluster talks in 2016, so this is one of the first. very well-received.
Operationalizing Redis At Scale
Brian Ip, Software Engineer, Square
– SSL everywhere with ghostunnel
– read-only using min clients 1000
– redis is ro-cache, but backups made for operational reasons like cache warming
– LXC environment
– looking forward to slides!
Building Large, High Performance Databases with Redis using Flash Memory and Emerging Hardware
Cihan Biyikoglu/Frank Ober
(Redis Labs/Intel)
– RAM is expensive, SSD is cheaper
– so use SSD in tiered storage where redis is aware
– Intel has Optane SSD series for this (gave one away to audience member)
How Roblox keeps millions of users up to date with Redis Pub/Sub
Peter Phillips, Senior Software Engineer
– crazy numbers – 1 million persistent chat connections using web tier, 4 redis nodes
– Windows environment
– try to keep connection count down to 1 per device but not easy
– pub/sub used for cache invalidation
– some attendees said “best technical talk”
Using Redis at scale at Twitter
Rashmi Ramesh, Senior Software Engineer
– LB proxy layer in front of redis
– online rebalancing for qps avg. 600k qps, not size
– nighthawk is internal name
– still use memcached for flat values
– patches or designs for identifying hot keys, recommends caching in app.
Thursday Keynotes
Doing More with Redis
Ofer Bengal and Yiftach Shoolman, Co-Founders, Redis Labs
Real-time intelligence with Spark and Redis
Reynold Xin, CTO, Databricks
– great talk using the Google NIPS 2015 machine learning diagram: LHS can be implemented Spark, RHS in Redis.
Microservices and Redis
Chris Richardson, Founder, Eventuate.io
– “microservices reflect business units”
– for distributed data modelling, look up Saga and CQRS
Chris is a world-class deep thinker. Check out his slide deck.
Trends in DevOps
Charity Majors, CEO, Honeycomb.io (ex-Parse, ex-Facebook)
Listening to Charity is definitely a unique experience. Very brassy, but also dead-on accurate.
– Redis is “Memory as a Service”
– monitoring vs. observability
– what’s changed?
– infrastructure and storage complexity is growing
– do you know who the DBA is? If not, it’s you! Image may be NSFW.
Clik here to view.
– distributed system: your system is never entirely up
– your green dashboard is a lie
– there are no more easy problems. there are only hard problems. (combinations, and you fixed the easy ones)
– context is everything. aggregates throw away data
– dashboards must be people-first and consumer-friendly
– don’t make everyone be an expert
– don’t promote swdevs who don’t know operations: #OwnYourProblems
Thomas Middleditch
(Star of the hit HBO show “Silicon Valley”)
Thomas is a professional comedian/actor who portrays an alpha nerd on TV. IRL, he is a gamer and did some QBasic programming as a kid. He fielded questions from the audience and was generally witty and entertaining. His TV persona is a composite of nerds, not inspired by one particular person. One of the funniest things he did was to read the conference poster buzzwords and exclaim, “I want to get me some more pam-auth.” Image may be NSFW.
Clik here to view.
Thursday Talks
Utilizing Redis in a High-tech Ad Traffic Stack
Rahul Babbar, Timesinternet (Advertising in Colombia, South America)
– 99% percentile is 2 ms
Best practices (also see slides):
– do BGSAVE on masters sequentially to reduce IO hotspot
– ensure TTL is set
– renamed destructive commands so that rogue client programs don’t have accidents
– set idle timeout
– hz patameter
– have an app strategy in case of redis slowdown or failure
Machine Learning in Real-time with Redis-ML
Shay Nativ, Redis Labs
– Redis’ plugins are a great way to add features
– one plugin is ML, with several models available
– one adtech user was able to use the tree model to reduce from 1200 Java app servers to 40 Redis servers
– slides
A Collaborative Canvas Using Bitfields in Redis
Daniel Ellis, Reddit
– about Reddit’s April Fools collaborative painting tool
– initial data model for Cassandra, not really a good fit
– changed to Redis using one key with 15 MB string representing one nybble for each pixel
Multi-Master Redis : A Deep Dive
Elad Ash, Redis Labs
– current Redis is just LWW (last write wins)
– for multi-master, it’s desirable to have some kind of write conflict strategy
– each data type is getting “vector clock-ized”
– implemented as a Redis plug-in
– available now for beta, to be released in 2018
Geofencing Using Redis Geospatial Queries
Matthew Hicks, Appboy
– Appboy library is installed on up to a billion mobile devices/apps, bundled with various apps
– use Redis built-in commands to calculate distance from an advertiser location to the current user location for ad targeting (circle diameter)
– For example “is a passenger approaching a branch location? if so, push an ad.”
– not difficult to implement using Redis, but need detailed user requirements to do so in a relevant way.
Geospatial Indexing At Scale: the 10 Million QPS Redis Architecture Powering Lyft
Daniel Hochman, Lyft
– good talk.
Getting There
From peninsula, take Caltrain to 4th and King then Muni to Powell Station, or BART from Millbrae Station to Powell Station.