[gentoo-user] How to synchronise between 2 locations

[J. Roeleveld]

Hi all,

I am looking for a way to synchronise a filesystem between 2 servers. Changes
can occur on both sides which means I need to have it synchronise in both
directions.

Does anyone have any thoughts on this?

Also, both servers are connected using a slow VPN link, which is why I can't
simply access files on the remote server.

--
Joost

[Mark Knecht]

Post by J. Roeleveld
Hi all,
I am looking for a way to synchronise a filesystem between 2 servers. Changes
can occur on both sides which means I need to have it synchronise in both
directions.
Does anyone have any thoughts on this?
Also, both servers are connected using a slow VPN link, which is why I can't
simply access files on the remote server.
--
Joost

How synchronized? For instance, does it need to handle identicals where
a file is on both sides but has been moved?

- Mark

[J. Roeleveld]

Post by J. Roeleveld

Post by J. Roeleveld
Hi all,
I am looking for a way to synchronise a filesystem between 2 servers.

Changes

Post by J. Roeleveld
can occur on both sides which means I need to have it synchronise in both
directions.
Does anyone have any thoughts on this?
Also, both servers are connected using a slow VPN link, which is why I

can't

Post by J. Roeleveld
simply access files on the remote server.
--
Joost

How synchronized? For instance, does it need to handle identicals where
a file is on both sides but has been moved?

Yes, it needs to handle changes to files/movements/deletions that occur on both
sides. Most of the time, there will only be changes on 1 side, but this is not
guaranteed.
I also need it to identify conflicts and handle this in a way that is
managable. (creating a copy with a different name would be sufficient)

--
Joost

[Grant Edwards]

Post by Mark Knecht

Post by J. Roeleveld
I am looking for a way to synchronise a filesystem between 2
servers. Changes can occur on both sides which means I need to
have it synchronise in both directions.

How synchronized? For instance, does it need to handle identicals where
a file is on both sides but has been moved?

Does it need to handle the case where the same file is modified
independently on both sides?

[J. Roeleveld]

Post by Grant Edwards

Post by Mark Knecht

Post by J. Roeleveld
I am looking for a way to synchronise a filesystem between 2
servers. Changes can occur on both sides which means I need to
have it synchronise in both directions.

How synchronized? For instance, does it need to handle identicals where
a file is on both sides but has been moved?

Does it need to handle the case where the same file is modified
independently on both sides?

Yes, as this is something that could happen

[ralfconn]

Post by J. Roeleveld
Hi all,
I am looking for a way to synchronise a filesystem between 2 servers. Changes
can occur on both sides which means I need to have it synchronise in both
directions.
Does anyone have any thoughts on this?
Also, both servers are connected using a slow VPN link, which is why I can't
simply access files on the remote server.
--
Joost

Unison. I use it just for that but can't say anything about the VPN bit,
my servers are on local network.

raffaele

[Frank Steinmetzger]

Post by J. Roeleveld
Hi all,
I am looking for a way to synchronise a filesystem between 2 servers. Changes
can occur on both sides which means I need to have it synchronise in both
directions.
Does anyone have any thoughts on this?
Also, both servers are connected using a slow VPN link, which is why I can't
simply access files on the remote server.

+1 for Unison. I’ve been using it for many years now to synchronise between
the four PC systems in my household.

I use it just for that but can't say anything about the VPN bit, my
servers are on local network.

Unison creates a local index of all files it syncronised. So when you move a
file around on one end, Unison will notice that because the file at the new
location has the same hash as the file at the old location. As a result, it
does not transmit the file anew to the remote host, but instead copies it
locally on the remote host.

Since Unison uses ssh underneath, you can use ssh’s transparent compression
to speed up the transfer.

[Alarig Le Lay]

+1 for Unison. I’ve been using it for many years now to synchronise between
the four PC systems in my household.
Unison creates a local index of all files it syncronised. So when you move a
file around on one end, Unison will notice that because the file at the new
location has the same hash as the file at the old location. As a result, it
does not transmit the file anew to the remote host, but instead copies it
locally on the remote host.
Since Unison uses ssh underneath, you can use ssh’s transparent compression
to speed up the transfer.

I’ve been thinking about using it to synchronise dovecot maildir folders,
since dsync is now deprecated. But I’m not sure about it as I never used
it under “heavy” loads. Do you have any thoughts about it?

[Frank Steinmetzger]

Post by Frank Steinmetzger
+1 for Unison. I’ve been using it for many years now to synchronise between
the four PC systems in my household.
Unison creates a local index of all files it syncronised. So when you move a
file around on one end, Unison will notice that because the file at the new
location has the same hash as the file at the old location. As a result, it
does not transmit the file anew to the remote host, but instead copies it
locally on the remote host.
Since Unison uses ssh underneath, you can use ssh’s transparent compression
to speed up the transfer.

I’ve been thinking about using it to synchronise dovecot maildir folders,
since dsync is now deprecated. But I’m not sure about it as I never used
it under “heavy” loads. Do you have any thoughts about it?

Among the many files I sync is my Mail folder. It currently contains around
106k files. My music collection is another 50k files. If there are no
changes and I did a recent run, then another run to look for changes takes
only very few seconds.

[J. Roeleveld]

Post by Frank Steinmetzger

Post by J. Roeleveld
Hi all,
I am looking for a way to synchronise a filesystem between 2 servers.
Changes can occur on both sides which means I need to have it
synchronise in both directions.
Does anyone have any thoughts on this?
Also, both servers are connected using a slow VPN link, which is why I
can't simply access files on the remote server.

+1 for Unison. I’ve been using it for many years now to synchronise between
the four PC systems in my household.

I use it just for that but can't say anything about the VPN bit, my
servers are on local network.

Unison creates a local index of all files it syncronised. So when you move a
file around on one end, Unison will notice that because the file at the new
location has the same hash as the file at the old location. As a result, it
does not transmit the file anew to the remote host, but instead copies it
locally on the remote host.
Since Unison uses ssh underneath, you can use ssh’s transparent compression
to speed up the transfer.

Unison sounds interesting. How does it handle conflicts (eg, file is changed on
both sides?)

--
Joost

[ralfconn]

Post by J. Roeleveld

Post by Frank Steinmetzger
Unison creates a local index of all files it syncronised. So when you move a
file around on one end, Unison will notice that because the file at the new
location has the same hash as the file at the old location. As a result, it
does not transmit the file anew to the remote host, but instead copies it
locally on the remote host.
Since Unison uses ssh underneath, you can use ssh’s transparent compression
to speed up the transfer.

Unison sounds interesting. How does it handle conflicts (eg, file is changed on
both sides?)

I use Unison GUI on one of the two machines (on the other peer it's just
a program invoked from the ssh). When the analysis is complete, the GUI
shows what it would do to sync the machines, indicating the conflicts
and giving you the chance to choose what to do.

I believe it can be used from the command line or maybe even in batch
mode instead of GUI but I never did it that way.

raffaele

[Frank Steinmetzger]

Post by J. Roeleveld

Post by Frank Steinmetzger
Unison creates a local index of all files it syncronised. So when you move a
file around on one end, Unison will notice that because the file at the new
location has the same hash as the file at the old location. As a result, it
does not transmit the file anew to the remote host, but instead copies it
locally on the remote host.
Since Unison uses ssh underneath, you can use ssh’s transparent compression
to speed up the transfer.

Unison sounds interesting. How does it handle conflicts (eg, file is changed on
both sides?)

I use Unison GUI on one of the two machines (on the other peer it's just a
program invoked from the ssh). When the analysis is complete, the GUI shows
what it would do to sync the machines, indicating the conflicts and giving
you the chance to choose what to do.
I believe it can be used from the command line or maybe even in batch mode
instead of GUI but I never did it that way.

You can set up a merge command to solve conflicts on the cmdline, such as
vimdiff. But when I set that, it blocks the GUI. Maybe I did something wrong
with the setup. Anyways, when I get a conflict, I make a backup of the file
locally, overwrite it with the remote and then do a conflict resolution with
vim.

In my every-day workflow, I usually only get conflicts in text files (logs,
notes, and so on). Binary conflicts are rare and usually due to recent
actions, such as editing an image or music file. In that case I can decide
on a per-case-basis which version to keep.

[Matt Connell]

Post by J. Roeleveld
Hi all,
I am looking for a way to synchronise a filesystem between 2 servers.
Changes can occur on both sides which means I need to have it
synchronise in both directions.
Does anyone have any thoughts on this?
Also, both servers are connected using a slow VPN link, which is why
I can't simply access files on the remote server.

I've been using syncthing for years and am extremely pleased with it.
It works so well that I sometimes forget that its there, truly in the
It Just Works category of software.

[Matt Connell]

Post by Matt Connell

Post by J. Roeleveld
Hi all,
I am looking for a way to synchronise a filesystem between 2
servers. Changes can occur on both sides which means I need to
have it synchronise in both directions.
Does anyone have any thoughts on this?
Also, both servers are connected using a slow VPN link, which is
why I can't simply access files on the remote server.

I've been using syncthing for years and am extremely pleased with
it. It works so well that I sometimes forget that its there, truly
in the It Just Works category of software.

Syncthing is also a good idea. The major difference: syncthing is a
permanently running daemon, so changes are synced very fast (the
interval is configurable, IIRC). OTOH, Unison is run individually by
you. That’s why I prefer the latter: in case I broke some file on my
machine, I can get it back from another machine without having to
break out the backup disk (which may not even have what I need
because my backup interval is too big).

Good point. I mainly use syncthing as a "stuff I need on multiple
machines" bucket, rather than a big directory of active working files.
Think more along the lines of "I saved this PDF on my PC but I'll need
it on my phone tomorrow so it goes into the Special Folder" kind of use
case. As such it might not be the right fit for "synchronise a file
system" use case.

[Frank Steinmetzger]

Post by Matt Connell

Syncthing is also a good idea. The major difference: syncthing is a
permanently running daemon, so changes are synced very fast (the
interval is configurable, IIRC). OTOH, Unison is run individually by
you. That’s why I prefer the latter: in case I broke some file on my
machine, I can get it back from another machine without having to
break out the backup disk (which may not even have what I need
because my backup interval is too big).

Good point. I mainly use syncthing as a "stuff I need on multiple
machines" bucket, rather than a big directory of active working files.

I sync most of my files on the home and media partition. So all my
documents, photos and music library. I do use syncthing -- between my
android phone and PC. Because I don’t like to fiddle with the filesystem on
the very constrained UI of touch devices.

[Frank Steinmetzger]

Post by Matt Connell

Post by J. Roeleveld
Hi all,
I am looking for a way to synchronise a filesystem between 2 servers.
Changes can occur on both sides which means I need to have it
synchronise in both directions.
Does anyone have any thoughts on this?
Also, both servers are connected using a slow VPN link, which is why
I can't simply access files on the remote server.

I've been using syncthing for years and am extremely pleased with it.
It works so well that I sometimes forget that its there, truly in the
It Just Works category of software.

Syncthing is also a good idea. The major difference: syncthing is a
permanently running daemon, so changes are synced very fast (the interval is
configurable, IIRC). OTOH, Unison is run individually by you. That’s why I
prefer the latter: in case I broke some file on my machine, I can get it
back from another machine without having to break out the backup disk (which
may not even have what I need because my backup interval is too big).

[tastytea]

Post by Matt Connell

Post by J. Roeleveld
Hi all,
I am looking for a way to synchronise a filesystem between 2
servers. Changes can occur on both sides which means I need to
have it synchronise in both directions.
Does anyone have any thoughts on this?
Also, both servers are connected using a slow VPN link, which is
why I can't simply access files on the remote server.

I've been using syncthing for years and am extremely pleased with
it. It works so well that I sometimes forget that its there, truly
in the It Just Works category of software.

Syncthing is also a good idea. The major difference: syncthing is a
permanently running daemon, so changes are synced very fast (the
interval is configurable, IIRC). OTOH, Unison is run individually by
you. That’s why I prefer the latter: in case I broke some file on my
machine, I can get it back from another machine without having to
break out the backup disk (which may not even have what I need
because my backup interval is too big).

syncthing has inbuilt backups if you enable it 😉

[J. Roeleveld]

Post by Matt Connell

Post by J. Roeleveld
Hi all,
I am looking for a way to synchronise a filesystem between 2 servers.
Changes can occur on both sides which means I need to have it
synchronise in both directions.
Does anyone have any thoughts on this?
Also, both servers are connected using a slow VPN link, which is why
I can't simply access files on the remote server.

I've been using syncthing for years and am extremely pleased with it.
It works so well that I sometimes forget that its there, truly in the
It Just Works category of software.

Syncthing is also a good idea. The major difference: syncthing is a
permanently running daemon, so changes are synced very fast (the interval is
configurable, IIRC). OTOH, Unison is run individually by you. That’s why I
prefer the latter: in case I broke some file on my machine, I can get it
back from another machine without having to break out the backup disk
(which may not even have what I need because my backup interval is too
big).

I had a quick look and it seems to depend on systems running outside of my
network. Can it be used without any link to a "centralised" server?

--
Joost

[tastytea]

Post by J. Roeleveld

Post by Matt Connell

Post by J. Roeleveld
Hi all,
I am looking for a way to synchronise a filesystem between 2
servers. Changes can occur on both sides which means I need to
have it synchronise in both directions.
Does anyone have any thoughts on this?
Also, both servers are connected using a slow VPN link, which
is why I can't simply access files on the remote server.

I've been using syncthing for years and am extremely pleased with
it. It works so well that I sometimes forget that its there,
truly in the It Just Works category of software.

Syncthing is also a good idea. The major difference: syncthing is a
permanently running daemon, so changes are synced very fast (the
interval is configurable, IIRC). OTOH, Unison is run individually
by you. That’s why I prefer the latter: in case I broke some file
on my machine, I can get it back from another machine without
having to break out the backup disk (which may not even have what I
need because my backup interval is too big).

I had a quick look and it seems to depend on systems running outside
of my network. Can it be used without any link to a "centralised"
server?

yes, you can disable the external servers. if the systems are running on
the same LAN (same VPN could work too?) they should find each other
nevertheless. or you could host your own discovery/relay server
somewhere.

[Grant Taylor]

Post by J. Roeleveld
Hi all,

Hi,

Post by J. Roeleveld
I am looking for a way to synchronise a filesystem between 2 servers. Changes
can occur on both sides which means I need to have it synchronise in both
directions.

What sort of turn around time are you looking for? seconds, minus,
hours, longer?

Post by J. Roeleveld
Does anyone have any thoughts on this?

I would wonder about using rsync.

host1 -> host2 at the top of the hour
host2 -> host1 at the bottom of the hour

Or if you wanted to get fancy

host1 pushes to host2 at the top of the hour
host2 pushes to host1 at a quarter past
host2 pulls from host1 at the bottom of the hour
host1 pulls from host2 at a quarter till

I'm thinking like if one of them was a road warrior and only one side
could initiate because of a stateful firewall.

Post by J. Roeleveld
Also, both servers are connected using a slow VPN link, which is why I can't
simply access files on the remote server.

ACK

[Frank Steinmetzger]

Post by J. Roeleveld
Hi all,

Hi,

Post by J. Roeleveld
I am looking for a way to synchronise a filesystem between 2 servers. Changes
can occur on both sides which means I need to have it synchronise in both
directions.

What sort of turn around time are you looking for? seconds, minus, hours,
longer?

Post by J. Roeleveld
Does anyone have any thoughts on this?

I would wonder about using rsync.

Post by J. Roeleveld
Also, both servers are connected using a slow VPN link, which is why I can't
simply access files on the remote server.

it would be beneficial to conserve traffic as much as possible.

While you can use the -u flag to only overwrite if the source is newer than
the destination, AFAIK rsync can’t detect if the destination has also been
altered since the last sync, so it might clobber important changes. That’s
why sync tools use a metadata cache to remember last edit timestamps.

[J. Roeleveld]

Post by J. Roeleveld
Hi all,
I am looking for a way to synchronise a filesystem between 2 servers.
Changes can occur on both sides which means I need to have it synchronise
in both directions.
Does anyone have any thoughts on this?
Also, both servers are connected using a slow VPN link, which is why I can't
simply access files on the remote server.
--
Joost

All,

Many thanks to everyone. I will be investigating and testing "net-misc/unison"
and "net-p2p/syncthing" to see which fits my requirement best.

Thanks again,

Joost