While Rsync is a suitable replication solution for copying or synchronizing smaller files from one Linux server to another, it (like most free command-line sync solutions) experiences occasional breakdowns and hard-to-diagnose replication errors. These errors force users and IT admins to spend precious time scouring forums in search of resolutions.
While there is no way for us to diagnose your Rsync error without taking a look at your system, this article will describe how you can check the status of your Rsync setup. Plus, we share several common solutions to potential Rsync errors.
We’ll also discuss our own file sync solution, Resilio Connect, and how it provides faster, more reliable replication.
Resilio Connect is an enterprise-proven file synchronization solution that uses P2P (peer-to-peer) file transfer and a UDP-based WAN acceleration transfer protocol to deliver fast (10 Gbps per server) file transfer and sync speeds over any network (including high-latency, lossy networks).
Resilio scales organically to handle large replication jobs and can sync large files and large numbers of files (tested at 250+ million files per job) to many endpoints in roughly the same time it takes to sync between two endpoints. Organizations in gaming (Blizzard, Bungie), tech (Match.com, Cisco), media (CBS, Turner Sports) and more use Resilio to reliably sync files across their replication environments and enhance business workflows.
To learn more about syncing with Resilio Connect, schedule a demo.
How to Check Rsync Status
There are two ways to check the status of your Rsync setup in order to diagnose and troubleshoot your failure.
1. Rsync Error Messages
When you run Rsync commands, one of two situations will occur:
- Rsync works properly
- Rsync returns an error message that indicates an issue
The following error message is an example of a message you might see if there’s an issue:
sh: rsync: command not found rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream These error messages provide valuable information that can help you narrow down your issue and avoid wasting time on erroneous solutions. For example, the error message above indicates that the error is due to a connection failure and not a flaw in your Rsync command syntax. So you should probe your Rsync setup to identify the connection issue rather than trying to alter your replication parameters.
2. Check Rsync Progress
If your error involves a specific sync job or specific files in your file system that aren’t replicating properly, then you should check your sync progress to find more information about where the error is occurring.
You can check Rsync progress using the following command:
rsync -- progress [source] [destination]Rsync will provide the following information about each individual file:
- Current transfer progress (as a percentage of transfer)
- File size (in bytes)
- Transfer speed (in bytes)
- Time until the transfer is finished
The display will look something like this:
File 113464738 100% 64.99 MB/s 0:02:00 (xfer#1, to-check=1/2)File 216253329 100% 85.92MB/s 0:02:00 (xfer#2, to-check=1/2)Check out our article on how to check Rsync progress for more information and commands you can use.
3 Common Solutions to Rsync Errors
Below, we share several common solutions to Rsync errors.
1. Install the Latest Version of Rsync
Rsync releases updates periodically to fix issues and debug their software. These updates can cause issues if you’re using an outdated Rsync version. For example, if you’ve already installed Rsync on a device, the new update can create conflicts that cause your sync environment to fail.
Updating to the latest version of Rsync will also make it easier to find and fix errors. Older Rsync daemons could not return errors generated during the option-parsing phase of the transfer. Upgrading to a newer daemon (at least 2.6.3) will allow such errors to get returned.
You can find the latest version of Rsync for Linux here and the latest version for Ubuntu here.
2. Ensure Rsync Is Installed on All Devices
This may seem too simple, but users often overlook the simplest steps. For example, a Rsync user in this forum didn’t realize that one of their remote machines didn’t have Rsync. Or Rsync may be on the path on the remote machine when running an interactive shell, but not when running Rsync by ssh with a non-interactive shell.
If the Rsync error message says “command not found”, this can be an indication that Rsync either isn’t on one of your devices or isn’t installed properly on one of your devices.
3. Double Check Your Rsync Syntax
The correct Rsync syntax should appear as follows:
Rsync [sync options/commands] [source dir] [destination dir]Your source device should come first and your destination device should come second.
Some common Rsync options include commands such as:
- -v: This command (which stands for verbose) commands Rsync to provide more information about what files are being transferred.
- -z: This command compresses file data.
- -r: This command performs a recursive copy of data but doesn’t preserve timestamps or permissions when transferring files.
- -a: This command copies data recursively (in archive mode) while preserving symbolic links, timestamps, file permissions, and user/group ownership.
4. Other Potential Solutions
Rather than create an overwhelming list of potential solutions to Rsync errors, we decided to share this list of curated forum questions and solutions:
- Rsync Not Working (using the IP address of the destination rather than the host name)
- Ask Ubuntu: Rsync Not Excluding a Folder When Commanded
- How to Use Rsync to Sync On Remote and Local Machines
- Rsync Not Working After Upgrading to LE10
- Rsync Suddenly Stopped Working On Linode (error code 255)
- Rsync Backup Not Working
5 Reasons Why Resilio Is a Superior Sync Solution
Rsync is a suitable sync solution for many small sync jobs and replication environments. But as your replication environment grows (i.e., syncing large files, large numbers of files, or too many endpoints), Rsync will experience more errors and breakdowns.
Resilio Connect is a superior file synchronization solution that was designed to handle large sync jobs. It uses P2P file transfer, real-time synchronization, and a WAN optimization protocol known as ZGT (Zero Gravity Transport™) to transfer files faster and more reliably across any network.
1. Resilio Is Easier to Use and Manage
Resilio is an agent-based software easy to set up and use. With its flexible BYO infrastructure model, it can be deployed on your existing infrastructure and begin replicating in as little as two hours.
Resilio can be deployed on premise, in the cloud, or in a hybrid cloud setup. It can be configured cross-platform (on Linux, Windows, Unix, Ubuntu, iOS, MacOS, Android, and more) and supports a variety of servers, desktops, virtual cloud servers, cloud storage platforms, and mobile devices.
Monitoring and controlling file replication in your environment is easy with Resilio’s Management Console. It provides real-time notifications, transfer logs, and more that make it easy to monitor replication progress and fix any issues that arise.
You also get granular control over how replication occurs, and can use the Management Console to:
- Adjust replication parameters, such as bandwidth usage, disk I/O threads, buffer size, and more.
- Set up Webhooks.
- Control user permissions.
Voicebase is a software company that provides speech-to-text for audio and video transcriptions. They use Resilio to distribute speech models (files of 50+ GB) to over 400 servers.
“Resilio Connect enables us to reliably distribute our code, specifically new language models in a fraction of time. These copy jobs now take an hour, down from eight. Best of all, once Resilio Connect was installed, it just works: We never need to manually intervene in any way.”
2. Resilio Provides Faster, More Reliable File Transfer and Sync
Resilio’s real-time sync capabilities and P2P transfer architecture provide some of the fastest sync speeds in the industry (we’re seeing up to 10 Gbps per server).
Tools like Rsync perform incremental syncs — i.e., syncing data at predetermined intervals (such as every 1 minute, 5 minutes, etc.). But Resilio syncs file changes in real time as they occur. It uses real-time notifications from the host OS and optimized checksum calculations (i.e., identification markers that change whenever a change is made to a file) to detect and synchronize file changes immediately.
Resilio’s P2P sync topology also outperforms Resilio’s point-to-point sync topology. While Rsync can only sync between two servers at a time (i.e., from one server to another), every server in a P2P environment is equally privileged and can sync simultaneously.
Resilio also uses a process known as file chunking, where a file is broken down into several chunks that can transfer independently from each other. This enables every device in your environment to engage in a sync simultaneously, drastically enhancing sync speed.
For example, if Server 1 wants to share a file with four other servers, it can split that file into three chunks. It can share the first chunk with Server 2. Server 2 can then share that first chunk with another server even before it receives the rest of the file. With all of the servers working together to sync files across your entire environment, Resilio can sync your files 3-10x faster than point-to-point solutions.
Resilio also implements state of the art security features, that were reviewed by third party experts, to protect your data and ensure safe delivery of your files. Resilio’s security features include:
- One-time session encryption.
- AES 256 encryption that encrypts data at rest and in transit.
- Mutual authentication to validate endpoints and ensure files are delivered only to approved devices.
- Cryptographic integrity validation to ensure that your files arrive intact and uncorrupted.
Larian Studios is a video game design company that uses Resilio Connect to distribute their video game builds across their geographically distributed offices.
“The nightly builds that we pump out involve a large amount of data. Getting that synced to every QA team in each studio proved to be quite difficult with the software that we had. With Resilio, as soon as our build servers have a build ready, it’s using the entire throughput of our gigabit lines to distribute it to the other studios. So QA can start in the morning on a fresh build.”
Learn more about how Resilio Connect helps Larian Studios move data 5x faster.
3. Resilio Can Easily Automate Sync Jobs
To automate sync jobs with Rsync, you’ll need to learn and implement a variety of Rsync commands and, in some cases, create complex setups. Identifying and resolving issues with these sync automations can be difficult.
But with Resilio Connect, you get all of the automation capabilities without any of the complexity. You can use command-line operations, the GUI, or Resilio’s REST API to easily automate any type of sync job.
4. Resilio Provides Superior WAN Transfer
Rsync uses transfer protocols that aren’t optimized for transfer over wide area networks. So if you need to sync files of high-latency, lossy networks, Rsync will perform poorly and slowly.
But Resilio uses a proprietary WAN acceleration protocol known as ZGT™ to optimize WAN transfer.
ZGT uses a congestion control algorithm to calculate the ideal send rate and create a uniform packet distribution over time. It also uses interval acknowledgements (acknowledgements that are sent periodically for a group of packets, rather than each individual packet) and retransmits lost packets once per RTT (Round Trip Time) to decrease unnecessary retransmissions.
ZGT can also dynamically route around network outages to ensure your files are delivered to their destination every time.
Shifo provides health data solutions that enable the delivery of healthcare services to over 3,000 remote communities worldwide. They use Resilio Connect to sync data to their servers in remote communities with poor network connections.
“You can’t imagine how difficult it was for us [prior to Resilio]: Delays, nothing being synchronized; now I don’t hear any of those problems. Now we can focus on solving healthcare problems [instead of moving files].”
Learn more about how Resilio Connect helps Shifo reliably sync data over the WAN.
5. Resilio Is Organically Scalable
As stated earlier, Rsync performs poorly when syncing large files, large numbers of files, or syncing to many endpoints. Since it syncs point to point, replication will take longer as your replication environment grows, and sync jobs can be delayed or stopped by network or hardware failures at any endpoint.
But Resilio’s P2P architecture means that it is organically scalable and resilient. Since every device can take part in synchronization, adding more devices (i.e., more endpoints) enhances replication speed. In a 1:2 replication scenario, Resilio will sync 50% faster than Rsync. In a 1:10 replication scenario, Resilio will sync 500% faster than Rsync.
P2P transfer also means that there is no single point of failure — if any device fails, the necessary files or services can be retrieved from any other device in your system. And Resilio can sync in any direction, including one-to-one, one-to-many, many-to-one, or N-way sync.
Reliably Sync Files with Resilio Connect
Resilio Connect is an enterprise-proven file synchronization solution that uses P2P file transfer and proprietary WAN optimization technology to quickly and reliably sync files across any network. It scales organically, and can sync large files/large numbers of files across hundreds of endpoints in roughly the same time that it takes to sync across two endpoints. And Resilio Connect provides an intuitive, user-friendly Management Console that makes it easy to control, monitor, automate, and troubleshoot replication across your entire environment.
To learn more about syncing with Resilio Connect, schedule a demo.
