Yesterday there was some great news in the Object storage industry. OpenIO announced it has broken, (although broken is not the right word here, as this was the first time that a test like this has been done) so let’s leave with that they set the record for write speed on an object store build on the Criteo platform. To achieve this OpenIO used the Criteo platform and deployed their software on a cluster with more than 350 physical servers. With this they exceeded a write speed of more than 1 Tbps (Terabits per second). In the end the maximum throughput reached a staggering 1373 Tbps. The official press release around this is added at the end of this post.
The TECHunplugged view
The object storage industry has always been seen for quantity instead of speed. And let’s be clear the sheer amount of data residing on object storage is all about the amounts, but there is as always need for speed. The record set by OpenIO and Criteo show that Object storage has more to offer than capacity alone, but more object storage vendors need to perform tests like these to really see the impact of what was shown in setting this record. At least OpenIO is giving us insight into what the product is capable of in terms of performance and scalability (amount of servers and not capacity) which are some amazing numbers. It is good news for the industry as a whole to see numbers like this, but we’re still very depending on the integration of all storage options in an IT environment. But knowing the object storage can have the performance showcased by OpenIO and Criteo is very promising.
The official press release
French Technology Startup Exceeds Write Speed of 1 Tbps Through Industry-First Storage Feat
Lille, France October 8, 2019 OpenIO, a leading provider of hyper-scalable, open source object storage today announced a record performance achievement with Criteo, the advertising platform for the open Internet. OpenIO demonstrated the performance and scalability of their object storage solution by deploying their technology on a cluster of more than 350 physical servers provided by Criteo. The benchmark made it possible to cross the symbolic threshold of writing up to one terabit of data per second, and even to exceed it since the useful throughput observed is 1.372 Tbps. This is the equivalent of digitally transferring the 22 million books from the world’s largest library, the Library of the U.S. Congress, in under one minute! OpenIO’s performance is a record because no other object storage technology has so far claimed and demonstrated such a high throughput and on such a large scale.
“Since the creation of OpenIO, we have placed the issue of performance at the very center of what we do,” explains Laurent Denel, CEO and co-founder of OpenIO. “Once we solved the problem of hyper-scalability, it became clear that data would be manipulated more intensively than in the past. This is why we designed an efficient solution, capable of being used as primary storage for video streaming (as is the case with our customer Dailymotion), or to serve increasingly large datasets for Big Data use cases.”
In recent years, due to the exponential growth in the volume of data collected by companies, their choices of storage technologies have often been guided by two criteria: the price per gigabyte and scalability – the ability to easily increase the capacity of a storage platform to create its “data lake” without creating data silos.
It is now possible to extract value from this data, and this extraction is becoming one of the main growth drivers for many companies. Also, the concern today is less to optimize the cost of archiving data than to make their processing possible by voracious Machine Learning and Deep Learning algorithms. After the era of Data Archiving and Data Sharing, we are entering the era of Data Processing.
#TbpsChallenge
The high performance that OpenIO achieved on Criteo machines under production conditions, confirms the design of the OpenIO object storage software for new data uses, in particular the massive exploitation of data by AI algorithms on Big Data / HPC clusters. Inaugurating the #TbpsChallenge with this record, OpenIO invites other market players to put their technology to the test (on commodity servers and tuned for production!).
“Criteo is pleased to have been a part of this benchmark, which gave OpenIO the opportunity to demonstrate how their technology can maintain consistent high performance while adding new nodes to scale massively” said Stuart Pook, Senior Site Reliability Engineer at Criteo. “They were able to achieve a write rate close to the theoretical limits of the hardware we made available.”
Criteo handles large volumes of data and has built a unique Big Data platform with several thousand nodes. Criteo’s team of engineers provided more than 350 machines from its infrastructure, recently racked in one of their datacenters, but not yet put into their production. An invaluable opportunity for OpenIO, because this cluster made up of commodity storage servers allows them to reach an order of magnitude that is truly ‘hyper-scale’.
About OpenIO
Founded in 2015, OpenIO develops a hyper-scalable, open source object storage solution that is ideal for Big Data, AI and HPC use cases. OpenIO’s grid of nodes architecture and unique ConscienceGridTM technology enable our object store to scale instantly without rebalancing data and while maintaining consistent high performance. OpenIO supports S3 and can be deployed on-premise, cloud-hosted or at the edge. It can run on any hardware and has already attracted more than 40 customers worldwide, which include among others, Dailymotion, the CEA, and the IIJ (Internet Initiative Japan) service provider.
OpenIO, which received initial support from Georges Lotigier (CEO of Vade Secure), raised $5 million from Elaia, Partech Partners and Nord France Amorçage in October 2017. In July 2019, the startup won the Pass French Tech 2018-2019, a national program to support hyper-growth companies. Based in Lille, France, OpenIO also has offices in Paris and Tokyo.
For more information please visit: http://www.openio.io
About Criteo
Criteo (NASDAQ: CRTO) is the advertising platform for the open Internet, an ecosystem that favors neutrality, transparency and inclusiveness. Close to 2,900 Criteo team members partner with close to 20,000 customers and thousands of publishers around the globe to deliver effective advertising across all channels, by applying advanced machine learning to unparalleled data sets. Criteo empowers companies of all sizes with the technology they need to better know and serve their customers. For more information