websiteger.blogg.se

This would allow them to scale their compute while scaling storage independently. First, it offered storage and compute separation backed by Amazon S3, an object store. It would also switch from Flipboard’s older data preparation and transformation platform to Apache Hive on Spark powered by Amazon EMR.įlipboard chose Amazon EMR for four primary reasons. Mactores would move Flipboard from its open-source hybrid data platform running on Amazon EC2 and host Apache HBase workloads on Amazon EMR backed by Amazon Simple Storage Service (Amazon S3). Following a detailed assessment of Flipboard’s core technology and a proof-of-concept, they agreed on a migration plan. “We lacked the expertise to design and architect it, and even if we had it, we didn’t have the bandwidth to deploy it.”ĪWS recommended Mactores, an AWS Advanced Consulting Partner specializing in data analytics and machine learning solutions. But the path to do that was unclear,” said Cris Favero, data engineering manager at Flipboard. “We wanted to upgrade to something more resilient and take much of the configuration management that open-source technologies require off our hands. But working with five-year-old versions of software and multiple data stores that ran on aging hardware created time-consuming hurdles for the engineering team. Initially storing its data in Amazon Elastic Block Store (Amazon EBS) volumes local to Amazon Elastic Cloud Compute (Amazon EC2) instances, it also used a third-party data lake platform. While the company is built on AWS, much of its environment relies on an open-source framework and applications, such as Apache HBase, to collect, process, analyze, and manage big data. Serving more than 100 million active users seeking curated content about every subject imaginable, Flipboard’s data environment sees up to 300,000 reads per second and 120,000 writes per second.