There’s basically 3 ways to store online data, where “storage” includes block and/or network locations:
- filesystems on top of blocks (zfs, xfs, ext4)
- object stores across storage (OpenStack Swift, Backblaze, S3)
- files erasure-encoded across storage networks (CleverSafe)
CleverSafe struggled along with VC funding until Oct. 2015, when it was bought by IBM for $1.3 billion.
An HN commenter has done us a favor by listing a handful of links to Intel CPU acceleration techniques useful for computing #3:
“At a glance, this seems like a clear explanation of using standard SIMD instructions to solve the problem, but I think the landscape has changed since this was written such that there are now better approaches.
In 2010, Intel released processors with a dedicated instruction for “packed carry-less multiplication.”
Unfortunately, the early implementations (through Sandy Bridge) were slow, and could be beaten by combining other SIMD operations as shown in this paper.
With the Haswell generation released in 2013, though, PCLMULQDQ got much faster. Instead of being able to complete one instruction every 8 cycles, it became possible to finish one every 2 cycles (inverse throughput went from 8 to 2). This 2015 paper “Faster 64-bit universal hashing using carry-less multiplications [PDF]” shows the difference this makes:
If you are looking for an explanation of how the problem could be solved with the basic building blocks of SIMD, the 2013 Plank, Greenan, Miller paper might be a good resource. But if you are hoping to implement high performance solution for modern processors, the 2015 Lemire and Kaser paper is probably a better starting point.
(This is with the caveat that I don’t actually understand the theory or terminology of Galois fields, and maybe there is something about applying it to Erasure Coding that makes the faster PCLMULQDQ approach inapplicable.)”
FAST-2013: Screaming Fast Galois Field Arithmetic Using Intel SIMD Instructions (2013) [PDF]
OpenStack Swift
register.co.uk: IBM Buys CleverSafe
Patent Troll Kills Open Source Project On Speeding Up The Computation Of Erasure Codes
Keywords: Reed, Solomon, Galois, GFC.