Hi all, quick question.
Given an FTP server and directory tree on local machine X and an FTP server on remote machine Y, I understand that a cron'd rsync script might be the best way to keep changes on X mirrored up to Y. My questions:
* Can I mirror the script in reverse so that changes on machine Y propagate back to X? (Perhaps running each script every other hour.)
* Every now and then, an extraordinary number of files need to be transferred, so a script running once an hour (or whatever) could conceivably bump into another instance of itself running. Issues?
For the curious, this is a temporary, partial workaround to my earlier bandwidth issues. By keeping a directory of local files updated remotely, we have the option of working on the files here [fast for us, miserable slow for outsiders who may want to get at them and work on them] or having freelancers work on them remotely [with faster access from our ISP-hosted FTP site]. Since work on the files is binary -- either a freelancer will work on them or we will, but not both -- having mirrored file trees seemed the way to go.
Many thanks, Greg
P.S. -- Congratulations to list member Nathan, who came and picked up the giant retro server yesterday. Nathan, thanks for freeing up approx. 100 square feet in my living room!
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Greg Brooks wrote: | Hi all, quick question. | | Given an FTP server and directory tree on local machine X and an FTP server | on remote machine Y, I understand that a cron'd rsync script might be the | best way to keep changes on X mirrored up to Y. My questions: | | * Can I mirror the script in reverse so that changes on machine Y propagate | back to X? (Perhaps running each script every other hour.)
Yes. Rsync has several options to let you synchronize just what you want. Update only files that already exist, update only files that are newer, etc. Check out the --update switch, which can probably be used to do what you want.
| * Every now and then, an extraordinary number of files need to be | transferred, so a script running once an hour (or whatever) could | conceivably bump into another instance of itself running. Issues?
This is typically handled using a lock-file (or something similar) so that only one instance is running at a time.
- -- Charles Steinkuehler [email protected]
Part A: You can push the changes both ways, but you are just asking for data loss if you don't have some sort of control on who can edit what files when. The first time you end up with two people modifying the same file on each server within the same cron window you will get to experience the issue. You will get the version with the latest time stamp, which may or may not be what you want. You will quite likely end up overwriting changes that you intended to have kept and not realize the mistake until much later.
Part B: Wrap your cron script in something like what I have below. It could be modified to touch a file remotely if you are trying to ensure that only one of the servers is running an instance at any given time.
#!/bin/bash if [ -f /some-where/rsync-is-running ]; then echo "`date` - An instance of rsync is already running!" >> /some-where/rsync.log else echo "`date` - Starting synchronization" >> /some-where/rsync.log touch /some-where/rsync-is-running # ... your script here ... echo "`date` - Finished synchronization" >> /some-where/rsync.log rm /some-where/rsync-is-running fi
Charles Steinkuehler wrote:
Greg Brooks wrote: | Hi all, quick question. | | Given an FTP server and directory tree on local machine X and an FTP server | on remote machine Y, I understand that a cron'd rsync script might be the | best way to keep changes on X mirrored up to Y. My questions: | | * Can I mirror the script in reverse so that changes on machine Y propagate | back to X? (Perhaps running each script every other hour.)
Yes. Rsync has several options to let you synchronize just what you want. Update only files that already exist, update only files that are newer, etc. Check out the --update switch, which can probably be used to do what you want.
| * Every now and then, an extraordinary number of files need to be | transferred, so a script running once an hour (or whatever) could | conceivably bump into another instance of itself running. Issues?
This is typically handled using a lock-file (or something similar) so that only one instance is running at a time.
_______________________________________________ Kclug mailing list [email protected] http://kclug.org/mailman/listinfo/kclug
If you're going to do this, you need to deal with the possibility that the script will die unceremoniously, leaving the lock file behind. This adds a trap, makes the lock file identify the machine, PID, and date/time the lock was asserted, and examines that information and attempts to determine whether the lock is really valid rather than blindly assuming that it is.
* #!/bin/bash Lock=**/some-where/rsync-is-running* *Log=**/some-where/rsync.log* *if [ -f $**Lock** ] then read <$**Lock** Machine PID RDate if [ `uname -n` = "$Machine" ] then if [ "x`ps -ocmd= -p $PID`" = "x$0" ] then echo "`date` - An instance of rsync is already running on this server: `ps -fp $PID`" >>$Log exit else echo "`date` Ignoring invalid lock entry: `cat **$**Lock**`" fi #[ "x`ps -ocmd= -p $PID`" = "x$0" ] else echo "`date` - An instance of rsync appears to be already running on $Machine: `cat **$**Lock**`" >>**$Log* * exit fi #[ `uname -n` = "$Machine" ] fi #**[ -f $**Lock** ] # If we make it this far, actually do it. **echo "`date` - Starting synchronization" >> **$Log**trap 'rm **$**Lock**; exit' 1 2 3 4 5 6 7 8 10 11 12 13 14 15 echo `uname -n` $$ `date` >**$**Lock* *# ... your script here ... echo "`date` - Finished synchronization" >> **$Log* *rm **$**Lock*
You could probably even use rcmd to validate the alleged PID of the process on the other machine, if you wanted to go to the trouble. Or just have one server be the designated initiator of the rsync, which simplifies things greatly.