Accelerating file transfer with Storage Made Easy and the File Fabric

Amid all the shifts in the world of digital media, one thing remains a constant: The need to be able to organize, index, search, access, transfer and process very large files and objects efficiently across a variety of platforms and services.

Every day, digital companies and broadcasters create thousands of large digital media files that are often stored across a mix of newer and legacy on-premises solutions, object storage, public and private clouds and SaaS applications.

However, being able to access and move these assets around efficiently, securely and accurately is vital as media production becomes more collaborative and global.

“FTP, FTPS and other legacy protocols are still often used in the media space. While securable, they cannot be accelerated as media assets continue to grow in scale,” said James Liddle, CEO of Storage Made Easy.

As cloud and on-site object storage are becoming increasingly appealing to digital media organizations and broadcasters, transitioning to these setups can introduce a potential round of complications and compatibility issues with less up to date solutions.

File Fabric for broadcasters

Storage Made Easy’s solution is already being used around the world across a myriad of industries, and it’s poised to become a key player in the broadcast industry.

“Although the product was not initially specifically designed for the media vertical, we found over time that it ticked a lot of the boxes that media and broadcast companies required but were finding it difficult to find in a single product,” said Liddle.

The File Fabric has already been implemented at a major North American media company as well as the U.K.’s Arsenal Media Group, the production arm of the popular Arsenal Football Club, notes Liddle.

One leading broadcaster initially tested the File Fabric’s M-Stream feature to restore data from Tier 2 Object Storage to Isilon. The results were a five to six times throughput increase compared to its existing solutions.

The Enterprise File Fabric was used along with 16 GB of RAM, 12 CPUs and a 10 GB network deployed on VMware vSphere 6.5 with 12 petabytes of raw data, which was encoded using erasure coding (ISA-L).

When running tests with a 1 TB file copying to and from Tier 2 object storage and Tier 1 Isilon, the company realized transfer speeds of 700 to 770 MBps, compared to single streams of data from object storage to Isilon at 100 to 150 MBps using the company’s existing suite of tools.

At Arsenal Media Group, the File Fabric was selected as its key tool for nonproprietary media content management.

The File Fabric was integrated with Arsenal’s on-premises and cloud object storage technology, giving editors, journalists and administrators an efficient “single pane of glass” view to find, use and collaborate on assets and projects.

Arsenal Media leverages the File Fabric in a variety of ways, including uploading camera footage from devices and into the editing workflow faster.

The group can also give external journalists access to assets thanks to the File Fabric’s secure content distribution functionality. Reporters can be given access to media files while Arsenal maintains access control with time-based access, password authentication and download limits.

Arsenal also integrated its existing media asset management manager, Dalet Flex, but with the added advantage of key File Fabric features available in the Flex MAM — all thanks to rich API.

File Fabric’s solution

These implementations leverage Storage Made Easy’s multi-cloud solution as a bridge between newer storage models and content management tools and their legacy predecessors.

The File Fabric provides media organizations with total flexibility in the choice of storage services by offering direct integration with over 60 backend storage protocols and APIs, including public cloud stores such as Amazon S3, Amazon Glacier, Dropbox, Google Drive, Azure as well as NAS, SAN Object Storage, StorNext local or network storage solutions.

If, however, a particular workflow or application requires integration with a legacy system, the File Fabric kicks in and legacy protocols such as WebDAV, FTP and SFTP can take over.

Once an asset has been indexed by the File Fabric, resources become seamlessly accessible to both users and applications, allowing them to search across all protocols in a single namespace via FTP, SFTP and WebDAV as well as web browsers and desktop or mobile clients or applications.

This allows teams from around the world to share storage resources and content, accessing and working with it concurrently, sharing it with others in the organization and collaborating on projects.

“End users are never bothered with the nuances of backend storage solutions, whether it be AWS S3, CIFS, NFS or any number of other options,” explained Doug Soltesz, Storage Made Easy’s director of product solutions.

Storage Made Easy’s Enterprise File Fabric is designed to address these challenges while also working seamlessly with the workflows demanded by large scale and distributed global media organizations that need to access, modify and store large objects and files but face integration challenges.

Indexing and content intelligence

The File Fabric is also able to index all existing content in place and without data replication. Once indexed, users or applications can search for assets across multiple storage providers either on premises or in the cloud, a key requirement for many media companies.

This is backed by the File Fabric’s content intelligence feature that enables it to extract metadata beyond just the filename, size and creation data. Instead, content intelligence can extract and index asset qualifiers such as quality, frame rate, codec and more.

The File Fabric can also be tied to cloud-based machine learning tools such as Google Vision, allowing further metadata extraction such as audio transcription and scene and object detection.

All of this metadata becomes searchable, and once a user finds the content or object they want to use, the File Fabric securely handles the movement of this asset. Users can move files between Tier 1 storage and nearline storage or share the asset with a broadcaster or distributor to syndicate or monetize it.

“This is something we again found that many broadcast and media companies had as a key product sourcing priority,” said Soltesz.

Protecting assets

Storage Made Easy also places priority on the security of the assets it transfers and indexes.

“File Fabric federates identity, file encryption and key management in a way that is transparent to end users,” said Liddle.

Data in flight between the File Fabric Server and end clients is protected with TLS 1.2 and current AES-256 encryption algorithms. Data transfers using legacy protocols such as FTP, with TLS, or SFTP, prioritize current ciphers.

Meanwhile, data at rest can be encrypted either using the File Fabric’s built in FIPS-certified encryption or, if available, by the storage’s own encryption technology such as Amazon S3 SSE and SSE-C.

Transferring assets

When it comes time to utilize resources, a global user interface lets users search and analyze metadata or access, play and transfer assets quickly and efficiently using the M-Stream parallel upload and download technology. This can make file moves up to 10 times as fast.

File transfer acceleration from Tier 1 to Tier 2 or archive storage rack to rack within or between data centers is also handled by the File Fabric, with integration between local storage solutions and public cloud stores.

Storage Made Easy also takes care of ensuring the files are moved in native formats and transferred over the proper protocols, preventing conversion into proprietary formats.

The File Fabric’s file transfer acceleration is also completely nonproprietary, allowing all data to be accessed or added, in a bi-modal fashion, either directly or through the File Fabric itself.

“File Fabric comes with ready-to-use scripts that media houses can bake into their workflow, just like they did with FTP and FTPS, but backed with Storage Made Easy’s powerful data acceleration — notably on the way to the storage — but also on the way down to a user’s desktop,” explained Soltesz.

Preserving the integrity of data transfers is equally important as making sure they are fast and efficient.

Storage Made Easy supports storing the MD5 sum of a file as a sidecar file on any backend storage, allowing for absolute verification of files and protecting against bit rot or provider corruption.

As a whole, the combination of Storage Made Easy’s M-Stream and the File Fabric solutions can provide media organizations and broadcasters with a broad range of tools that accelerate asset and object management, discovery and transfers while also fitting in to existing workflows and legacy systems.

 

The article above is sponsor-generated content from Storage Made Easy. To learn more about sponsor-generated content, click here.