Efficient Object Storage Gateway Solution: Resilio Connect

Samuel Bennett

Home » Efficient Object Storage Gateway Solution: Resilio Connect

Resilio offers a turnkey storage gateway—specifically a “file gateway” to popular object storage systems from a variety of vendors. The Resilio file gateway works with any type of object storage: on-prem or in the cloud.

Object storage gateways provide companies with durable, long-term, and low-cost storage. Companies that have remote or distributed workforces and work with large amounts of data (i.e., large files, large numbers of files, or both) should seek an object storage gateway solution that allows them to access storage from anywhere—and one that makes object storage as easy to use as files and folders on a desktop. It’s also a great way for companies to seamlessly integrate object storage into their file-based workflows while also optimizing cloud storage costs.

Resilio Connect offers a file gateway that provides a file interface for objects stored in S3 and other object stores. Through Resilio Connect, all objects are accessible simply as files and folders.

There are two models for using Resilio Connect as a file gateway:

As a central gateway in the office or a remote site.
Directly from end-users’ workstations and desktops.

Using this model, users can connect to object storage from any location, globally. In the gateway model, all users connect to the gateway first in order to access object storage.

Resilio supports both on-premises and cloud-based object storage. Resilio also enables you to extend your other on-premises storage (direct-attached, SAN, or NAS) to any S3-compatible cloud storage platform (such as Amazon s3, Wasabi, Minio, Azure, and more) and transform your on-prem servers and NAS devices into flexible storage gateways.

You can use Resilio Connect to remotely browse, selectively sync, and manage all of your cloud and on-prem storage from a single, unified location. And Resilio Connect was designed to provide efficient local caching and low-latency access to data stored in the cloud, so you can minimize cloud storage costs (such as data egress fees) while maximizing productivity — making it an ideal solution for hybrid cloud, multi-cloud, and remote/distributed workforce scenarios.

In this article, we’ll discuss how Resilio Connect acts as an object storage gateway solution. We’ll cover how it helps companies:

Cache and access files efficiently (and minimize cloud storage costs): As a software-only system, Resilio eliminates the need to invest in expensive hardware (such as dedicated file servers for each office) and delivers global file accessibility directly on each local endpoint. You can store frequently used files locally and store infrequently used files in long-term cloud file storage.

Optimize workflows through flexible synchronization: Resilio enables you to fully control when files are synchronized and which endpoints they’re synchronized to. You can also create policies that selectively store, download (or partially download), and sync files as needed.

Unify data management from one vendor-agnostic platform: Resilio is a vendor and hardware-agnostic solution that can be deployed on your existing IT infrastructure with minimal expense and interruption to your operation. You can manage all of your on-prem and cloud data storage from one unified location — making it an ideal solution for hybrid and multi-cloud environments.

Quickly synchronize files over any network: Resilio’s P2P (peer-to-peer) synchronization architecture and WAN acceleration technology enable you to organically scale your replication environment, sync files of any size or number (we’ve tested and successfully synced 450+ million files in a single job) 3-10x faster than traditional solutions (up to 20+ Gbps per server), sync files in any direction (one-way, bidirectional, one-to-many, many-to-one, and N-way), and transfer files over any network (including high-latency, lossy, and consumer-grade networks).

Protect your data with state-of-the-art, native security features: Resilio’s native encryption and data protection features keep your data safe and eliminate the need to invest in 3rd-party security tools (such as VPNs).

Organizations in media (Skywalker Sound, CBS), gaming (Blizzard, 2K Games), retail (Exxon Mobil, KFC), tech (Microsoft, Cisco) and more use Resilio Connect for hybrid and multi-cloud storage. To learn more about using Resilio Connect as your object storage gateway solution, schedule a demo.

Cache and Access Files Efficiently (and Minimize Cloud Storage Costs)

Resilio Connect is designed to deploy and operate efficiently, enabling you to reduce operational costs associated with cloud object storage and data management.

As a software-only, vendor-agnostic solution, you can install Resilio Connect on your existing IT infrastructure (without investing in expensive hardware) and begin replicating and accessing your data in as little as 2 hours. Your team can continue using the tools you’re already using without the need to learn or operate new systems.

Resilio’s file gateway enables you to extend your on-premises applications to the cloud. You can store frequently accessed files on local storage, while storing infrequently accessed files in long-term cloud storage — freeing up space on your on-prem servers. End-users can browse files remotely, then choose only the files they want to sync and download (using Server Message Block, or Network File System protocols).

While some hybrid cloud infrastructure solutions can only provide file access through SMB or NFS (which limits performance and necessitates the added expense of having a dedicated file server in each office), Resilio’s P2P replication architecture delivers global file access directly on each local endpoint. This minimizes costs while enhancing productivity, making it the ideal solution for high-performance workflows with large datasets and enterprise applications.

Resilio also minimizes costs by:

Providing native data security, eliminating the need to invest in 3rd-party security tools.
Enabling you to manage all of your on-prem and cloud data from one unified location, eliminating the need to invest in and use multiple cloud storage gateways in multi-cloud scenarios.

Optimize Workflows Through Flexible Synchronization

Cloud storage facilitates effective collaboration among remote and distributed workforces. But delays in how files are synchronized and accessed by employees can hinder operational efficiency.

Resilio Connect enables you to automate and optimize collaboration by giving IT teams and end-users more control over synchronization and file access.

Through a feature known as Transparent Selective Sync, you can selectively download, store, and sync files as needed. In other words, you can create policies that govern which files are synchronized to which endpoints, as well as how files are downloaded, locally cached, and purged. End-users can download, partially download, store, and sync files as needed — giving them quicker access to the files they need when they need them.

Operations Team: Resilio Connect Management Console

This capability is ideal for remote work scenarios that require granular access over how and where files are synchronized. With Resilio, you can sync in real-time or on a set schedule. And rather than syncing all files across your entire system, you ensure files are only synchronized to select endpoints (such as in situations where only privileged employees should be given access to privileged files).

Vendor-Agnostic, Unified Data Management

Resilio Connect is a vendor and hardware-agnostic solution that you can use with:

The servers, networks, desktops, DAS, NAS, and SAN storage volumes your team is already using
Popular operating systems, such as Windows, Linux, Mac, Android, iOS, and more (Resilio also offers iOS and Android apps)
Any cloud service, such as Wasabi, AWS cloud, Azure Blobs, Google Cloud Platform, MinIO, Backblaze, and more
VMware, Citrix, hypervisors (such as Microsoft Hyper-V), and other virtual machines

Resilio’s flexibility makes it easier (you can continue to use the tools and hardware you’re already using), quicker (you can deploy and operate Resilio in as little as 2 hours), and more cost-effective (there’s no need for data migration or to invest in new hardware) to deploy.

It also means that, rather than using multiple object storage gateways for different cloud storage services, you can manage your entire cloud and on-premises data infrastructure from one unified location — saving you money and simplifying your operation.

From Resilio’s Management Console, you can:

Create automated policies that govern bandwidth allocation for each endpoint. You can even create profiles for each endpoint that govern how much bandwidth it can use at certain times of the day and on certain days of the week.
Create and manage replication jobs.
Deploy instructions across public, private, and hybrid cloud storage.
Manage S3 objects and buckets.
Manage and monitor agents and job functions.
Adjust replication parameters, such as buffer size, disk I/O, and more.
Collect logs and configure notification events to be sent to email or Webhooks.

You can use Resilio’s powerful REST API to integrate applications and matching frameworks, as well as configure any type of scripting and automation your workflow demands.

End-users can browse and access files from a unified interface that operates much like Microsoft OneDrive — so everyone has the same views of files and folders.

How to select the "Always keep on this device" option.

Quickly Synchronize Files Over Any Network

Resilio Connect was designed to give you access to high-performance synchronization, so you can quickly sync files of any size or number over any network.

Optimized WAN Transfer with Proprietary WAN Acceleration Protocol

Most file replication solutions use inefficient transfer protocols that weren’t designed for WAN transfer.

For example, TCP/IP, a commonly used protocol, reduces the rate of data transfer when it detects too much packet loss. Since packet loss occurs in varying degrees over WANs, this isn’t a particularly effective method for dealing with such an issue.

But Resilio Connect uses a proprietary WAN acceleration protocol, known as Zero Gravity Transport™ (ZGT), which optimizes how data is transferred across WANs by:

Using a congestion control algorithm that constantly probes the RTT (Round Trip Time) in order to detect and compute the optimal rate of data transfer.

Sending packet acknowledgements in intervals, rather than after receiving each packet, to increase transfer speed.

Using delayed retransmission, which retransmits lost packets in groups periodically (in order to save time and increase transfer speed).

ZGT enables Resilio to optimize data transfer over any network, including high-latency, lossy, and consumer-grade networks. It also enables Resilio to sync data at the edge of networks, where there may be little to no network coverage (such as in remote locations, at sea, etc.)

Fast, Omnidirectional Synchronization in Real-Time

Resilio Connect delivers some of the fastest replication speeds in the industry (up to 20+ Gbps per server) using real-time replication, a P2P transfer architecture, and file chunking.

Using notification events from the host operating system and optimized checksum calculation (i.e., identification markers that change whenever a change is made to a file), Resilio can immediately detect and synchronize only changed portions of a file.

Other replication solutions use inefficient replication topologies that reduce sync speed and create single points-of-failure — such as hub-and-spoke replication (where files must first be transferred to a hub server, which then replicates the files to the other servers one at a time) or follow-the-sun replication (where replication occurs from one server to the next sequentially).

But Resilio Connect utilizes a P2P transfer architecture. In this replication topology, every server can share files with any other server simultaneously.

When syncing a file, Resilio uses a process known as file chunking to split the file into several chunks that can transfer independently of each other. As soon as a server receives a file chunk, it can begin sharing that chunk with any other server, even before it has received the full file (as demonstrated in the image below).

Through the combination of file chunking and P2P transfer, you can utilize the full bandwidth of your entire replication environment and achieve sync speeds 3-10x faster than traditional solutions.

Resilio’s P2P architecture enables you to scale out omnidirectional replication (you can sync in any direction, including one-way, bidirectional, one-to-many, many-to-one, and N-way sync) to quickly replicate data across cloud regions and on-premises endpoints. Resilio’s synchronization uses push and pull caching, so you can push updates across your entire environment in real-time and pull remote files on-demand.

Plus, P2P replication means that your system has no single point-of-failure. If any server or network goes down, the necessary files or services can be retrieved from any other server in your system — making Resilio an ideal solution for hot site disaster recovery use cases.

Hot/Live DR: Multi-site Active/Active; Warm DR: Active/Active; Cold DR: Active/Passive; Offsite Copy: Backup Copy

Protect Your Data with State-of-the-Art, Native Security Features

Many file gateways and replication solutions don’t include security features, forcing you to invest in 3rd-party security tools and VPNs.

But Resilio saves you the expense and keeps your data safe using native security features that include:

End-to-end encryption: Resilio encrypts data at rest and in transit using AES 256.

Data immutability: Resilio stores immutable copies of files in the public cloud, protecting you from data loss and ransomware.

Cryptographic data integrity validation: Using data integrity validation, Resilio ensures your files always arrive at their destination intact and uncorrupted.

Mutually authenticated endpoints: Before initiating a transfer with any endpoint, that endpoint must provide an authentication key.

Access control: Resilio’s Management Console enables you to control user permissions on specific files and folders, and grant access only to approved employees.