Organizations that use Linux computers to collaborate on projects, synchronize company data, or distribute web and app content updates need a server replication solution that can quickly and reliably replicate files/file changes across multiple endpoints using a variety of systems (physical, virtual, containerized, or cloud-based).
Most Linux file sync solutions (such as rsync) suffer from performance issues that can impede company workflows. These include issues, such as:
- Lack of WAN optimization: Without WAN acceleration technology, replication and transfer speed are slow and replication is unpredictable — particularly across high-latency WAN networks.
- Lack of scalability: Most solutions have difficulty syncing large files or large numbers of files.
- Limited to one-way sync: Most solutions can only sync in one direction, and can’t perform bidirectional, one-to-many, many-to-one, or many-to-many sync.
- Lack of management and monitoring: Many solutions provide no central way to monitor, track, and control replication across multiple endpoints.
- Lack of reliability: Replication failures and breakdowns can be difficult to diagnose and fix.
- Lack of extensibility: While most Linux-based tools support scripting from the command-line, most do not provide other forms of automation or integration through APIs.
In this article, we’ll share 5 high-performance server replication solutions that can be used to reliably synchronize files across Linux servers. We’ll begin with a deep-dive into our own server replication software, Resilio Connect.
Resilio Connect is an agent-based file replication solution that uses a P2P transfer architecture and WAN acceleration to deliver the fastest replication speeds in the industry (10+ Gbps) over any network. Because it replicates using a mesh network, Resilio can simultaneously sync files in any direction (one-to-one, bidirectional, one-to-many, N-way sync) and is particularly well-suited to large replication deployments (i.e., syncing large files, syncing large numbers of files, and syncing to many endpoints).
Top organizations in gaming (Larian Studios, Blizzard, Wargaming.net), tech (VoiceBase, Match.com, Kaspersky), media (CBS, Turner Sports, Skywalker Sound), and more use Resilio Connect to synchronize data and enhance workflows.
To learn more about how Resilio Connect can quickly and reliably replicate files across your Linux devices, schedule a demo.
1. Resilio Connect
Resilio Connect is a real-time server replication software that can be used for synchronizing, consolidating, and distributing files across Linux servers for disaster recovery, web content replication, cloud file storage, remote work for distributed workforces, and more.
It’s a flexible solution that provides users with the option to run a variety of hardware — such as x86, ARM, etc. — and your distribution of choice — such as Ubuntu, CentOS, etc. (See full Resilio system requirements here.)
Because it uses a peer-to-peer replication topology, Resilio has no single point-of-failure and can fully utilize network bandwidth across your entire environment to enhance replication speed and reliability. Connect can sync 250+ million files per job, replicate to tens of thousands of endpoints, and update hundreds of devices in roughly the same time it takes to update a single server.
Resilio Connect is the best solution for syncing files across Linux devices because it:
- Uses a P2P transfer architecture to provide fast, reliable, and organically scalable synchronization.
- Uses a proprietary WAN optimization protocol to enhance file transfer over WANs (wide area networks).
- Makes it easy to deploy, manage, and automate file synchronization.
- Is easily automated through scripting before or after jobs, as well as a fully extensible and well-documented API set.
- Works with a variety of open source tools and frameworks, from management tools like Grafana to a full range of DevOps tools.
Fast, Organically Scalable Replication with P2P File Transfer
To understand why Resilio Connect is superior to other replication solutions, you must first understand the difference between Resilio’s P2P replication topology vs. the point-to-point replication topologies — either the client-server topology or the “follow-the-sun” topology — used by most other solutions.
Let’s assume that you want to synchronize files across 10 servers in different locations.
In a client-server topology, one server is designated as the hub-server, and the other nine servers are designated as client devices. The hub-server can share files with any of the client devices, but the client devices can only share files with the hub-server (not each other). For example, in order for Client Device 2 to synchronize its files/file changes with the other devices, it must first send those changes to the hub-server, which will then replicate those changes to each of the other client devices one-by-one.
Replication in a “follow-the-sun” topology occurs from one server to another sequentially. In other words, Server 1 syncs with Server 2; Server 2 then syncs with Server 3; Server 3 with Server 4; and so forth.
But in Resilio’s P2P replication topology, every server can communicate with every other server. P2P file transfer is key to Resilio’s ability to outperform other solutions because it:
- Synchronizes files in real-time
- Replicates across devices simultaneously
- Utilizes the full bandwidth of your network to eliminate bottlenecks and single points-of-failure
- Scales organically to optimally handle large deployments
Real-Time File Synchronization
Many replication solutions rely on incremental synchronization. In other words, they periodically scan your entire file directory in order to identify and replicate file changes and new files. If you’re syncing large files or a large number of files, system scans can take a long time. And if your application depends on all servers having the most up-to-date versions of files, this model can be unreliable.
Resilio, however, uses optimized checksum calculations and notifications from the host OS to identify and replicate file changes as they occur. A checksum is an identification marker for a file that changes whenever a file is updated — alerting Resilio to the fact that a new file change needs to be replicated.
By replicating only changed portions of a file in real-time, Resilio can synchronize your servers faster and ensure your files are always where they need to be.
Simultaneous Replication
When replicating a file, Resilio Connect uses a process known as file chunking. The file is split into several blocks that can transfer independently from each other, enabling Resilio to sync across devices simultaneously.
For example, let’s assume there are ten servers in your replication environment, and Server 1 needs to sync a file with the other nine servers. Resilio would split that file into several blocks, then share it with Server 2. As soon as Server 2 receives the first file block, it can begin sharing it with Server 3, even before it receives the full file. And Server 3 can begin sharing that file block with any other server immediately after receiving it.
Since every server can share file blocks with every other server, Resilio can sync files across your environment 3-10x faster than traditional solutions.
No Bottlenecks or Single Points-of-Failure
In a client-server model, replication is entirely dependent on the hub-server. If the hub-server goes down, your application fails. This forces organizations to spend large amounts of money on backup servers and complex failover systems.
And in both point-to-point topologies, any server can become a bottleneck in your replication workflow. Any replication issues (such as replication failure or replication delays caused by slow networks) between devices can delay synchronization across your environment, as the other servers must wait for the issue to be resolved before they can receive files.
But since Resilio replicates using a P2P mesh network, there are no single points-of-failure. If a device goes down, the necessary files or services can be received from any other device in your environment. Resilio can dynamically route around failures and slow networks to ensure synchronization occurs as quickly as possible.
Organic Scalability & Omnidirectional Sync
Synchronizing large files, synchronizing large numbers of files, and/or synchronizing to many endpoints takes a long time when replicating point-to-point. The more servers you add to your environment, the longer it will take to synchronize files across your entire environment.
But Resilio was designed to specifically handle large, enterprise deployments, and has been tested to replicate 250+ million files in a single replication job.
Since all of your devices can share files with each other simultaneously, adding more endpoints actually enhances performance. All of the servers can take part in replication, and Resilio can balance the load across your entire environment to avoid overworking any single network or device.
Resilio can sync files 50% faster than a point-to-point solution in a 1:2 transfer scenario. In a 1:10 scenario, synchronization will occur 500% faster.
Resilio can also sync files in any direction required — e.g., one-to-one, two-way, one-to-many, many-to-one, many-to-many (aka N-way sync). This is particularly important when you need to synchronize files across many (hundreds or thousands) of endpoints quickly, such as syncing web/app servers, distributing software updates, and synchronizing files among remote, geographically distributed workforces.
Case Study: Larian Studios
Larian Studios is a video game development company that uses Resilio Connect to synchronize nightly game builds across their offices. “For us, the main thing was fast transfer of data and obviously maintaining the integrity of all the data that we sync to our offices. And definitely the cost is very important. We were happy to find a solution where we could use our in-place architecture to keep down the cost.”
Read more about how Larian Studios used Resilio to increase file sync speed by 500%.
Zero Gravity Transport™ Protocol for Optimized WAN Transfer
Resilio Connect uses a proprietary WAN acceleration protocol called Zero Gravity Transport™ (ZGT) to transfer files across high-latency WANs (wide area networks).
Most file replication solutions use transfer protocols that work well on LANs (local area networks) but perform poorly over WANs, such as TCP/IP. When transferring files over a WAN, varying degrees of packet loss can naturally occur. When packet loss occurs, TCP/IP treats it as a network congestion issue and reduces transfer speed in order to ease the burden on the network. Since packet loss during WAN transfer isn’t due to network congestion, this is a poor strategy and results in longer transfer times.
But ZGT is a transfer protocol that optimizes WAN transfer. It uses a congestion control algorithm to avoid overburdening the network by calculating the ideal send rate and creating a uniform packet distribution over time. It also increases transfer speed by:
- Using interval acknowledgements for a group of packets (rather than acknowledging each individual packet) that also provides information about lost packets.
- Retransmitting lost packets once per RTT (Round Trip Time).
Case Study: VoiceBase
VoiceBase is a speech-to-text software company that uses Resilio to regularly distribute speech model files of 50+ GB across 400 servers. “Resilio Connect enables us to reliably distribute our code, specifically new language models in a fraction of time. These copy jobs now take an hour, down from eight. Best of all, once Resilio Connect was installed, it just works: We never need to manually intervene in any way.”
Learn more about how Resilio Connect enabled VoiceBase to reduce replication time by 88%.
Easy to Deploy, Manage, and Automate File Synchronization
Unlike many other server replication solutions that use command-line interfaces, Resilio Connect is designed to be flexible and easy to deploy/use.
Resilio Connect operates by installing agents on your replication devices. You can deploy Resilio on your existing infrastructure and begin replicating in as little as 2 hours.
Resilio Connect is also incredibly flexible. It can:
- Be deployed on-premises, in the cloud, or in hybrid cloud environments.
- Support a variety of servers, mobile devices, virtual cloud servers, cloud storage services, and desktops.
- Operate cross-platform on Linux, macOS, iOS, Android, and Microsoft devices.
Resilio provides a centralized Management Console that makes it easy to control, manage, and automate replication jobs. You’ll get insight into the state of replication in your environment through real-time notifications and detailed replication logs. And you can control and automate how replication occurs in your environment by:
- Configuring rules and notifications
- Setting up Webhooks
- Controlling user permissions
- Adjusting replication parameters (e.g., buffer size, disk I/O threads, bandwidth, and more)
- Using Resilio’s REST API to script and automate replication jobs
Resilio Connect also protects your files using security features that were reviewed by 3rd-party security experts. Files are encrypted end-to-end using AES 256. And Resilio uses mutual authentication to ensure data is only delivered to designated endpoints and cryptographic data integrity validation to ensure files arrive at their destination uncorrupted.
To learn more about using Resilio Connect to sync files across your Linux devices, schedule a demo.
2. Suresync
SureSync is a server replication solution that works on Windows, Unix, Linux, Netware, and NAS devices. While it speeds up file transfer using remote differential compression to only copy changed portions of files, it uses TCP/IP transfer protocol —making it a poor solution for WAN transfer.
Suresync provides an Enterprise Status panel that you can use to monitor and control replication jobs and create user permissions.
3. Syncthing
Syncthing is the only other P2P software for syncing files across Linux devices. Syncthing can be installed on Mac OS X, Linux, FreeBSD, OpenBSD, Windows, and Solaris devices.
Syncthing protects data using TLS encryption and cryptographic certification. It provides an open protocol and source code that you can use to control and optimize your file replication (though this comes with a steep learning curve for new users).
4. GoodSync
GoodSync is a file synchronization solution that works on Linux, Windows, iOS, Android, Mac, and NAS devices. It offers a free and paid version, making it one of the most cost-efficient options on this list.
Goodsync speeds replication by syncing data in parallel threads. It also encrypts data using AES 256 and protects against data loss using bit-by-bit synchronization.
5. Rsync
Rsync is a command-line software that synchronizes files across Linux and Unix devices. It uses a remote update protocol to only replicate changed portions of files in order to speed up file transfer and end-to-end checks to ensure file transfer occurs and files reach their destination.
While Rsync is fairly reliable for simple transfer jobs and simple replication environments, it suffers from limitations such as:
- Rsync provides no native encryption.
- Rsync won’t transfer open files.
- Rsync experiences errors when synchronizing a large number of files.
Synchronize Linux Servers with Resilio Connect
Resilio Connect is the best solution for syncing your files on Linux devices because it is:
- Fast: Resilio’s P2P architecture enables it to quickly sync files across your environment in real-time.
- Reliable: With P2P transfer, there are no bottlenecks in your workflow. And Resilio can dynamically route around network outages.
- Optimized for WAN transfer: Resilio uses proprietary WAN optimization technology to enhance file transfer over WANs.
- Organically scalable: Resilio’s performance grows organically along with your replication environment. It’s the best solution for quickly syncing large files or large numbers of files across many endpoints.
- Secure: Resilio provides state-of-the-art, end-to-end encryption to protect your data at rest and in transit.
- Easy to use: Resilio can be deployed quickly on your existing infrastructure. And you can easily manage replication with Resilio’s Management Console.
To learn more about syncing Linux devices with Resilio Connect, schedule a demo.
