<feed xmlns='http://www.w3.org/2005/Atom'>
<title>admin/pylsyncd</title>
<subtitle>Python implementation of lscynd with performance in mind</subtitle>
<link rel='alternate' type='text/html' href='http://labs.feurix.org/admin/pylsyncd/'/>
<entry>
<title>Add notes on performance</title>
<updated>2009-11-25T19:51:38Z</updated>
<author>
<name>John Feuerstein</name>
<email>john@feurix.com</email>
</author>
<published>2009-11-25T19:51:38Z</published>
<link rel='alternate' type='text/html' href='http://labs.feurix.org/admin/pylsyncd/commit/?id=1ea2c10fda773bfcf932061287b0dffb9d38e4b3'/>
<id>1ea2c10fda773bfcf932061287b0dffb9d38e4b3</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Add proper error handling to TODO list</title>
<updated>2009-11-25T19:12:53Z</updated>
<author>
<name>John Feuerstein</name>
<email>john@feurix.com</email>
</author>
<published>2009-11-25T19:12:53Z</published>
<link rel='alternate' type='text/html' href='http://labs.feurix.org/admin/pylsyncd/commit/?id=5b157ce3e77adc4ba03699d9442cd66800e517c0'/>
<id>5b157ce3e77adc4ba03699d9442cd66800e517c0</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Fix rsync destination paths in README</title>
<updated>2009-11-25T19:11:49Z</updated>
<author>
<name>John Feuerstein</name>
<email>john@feurix.com</email>
</author>
<published>2009-11-25T19:11:49Z</published>
<link rel='alternate' type='text/html' href='http://labs.feurix.org/admin/pylsyncd/commit/?id=79551963cf9e5a943025909acb0b41a8a10fff2d'/>
<id>79551963cf9e5a943025909acb0b41a8a10fff2d</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Add quickstart notes to README</title>
<updated>2009-11-25T19:10:04Z</updated>
<author>
<name>John Feuerstein</name>
<email>john@feurix.com</email>
</author>
<published>2009-11-25T19:10:04Z</published>
<link rel='alternate' type='text/html' href='http://labs.feurix.org/admin/pylsyncd/commit/?id=2714f313e6b08e364b136b49b8e6dbd46d70977d'/>
<id>2714f313e6b08e364b136b49b8e6dbd46d70977d</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Added a few more comments and updated the TODO list slightly.</title>
<updated>2009-11-25T18:02:13Z</updated>
<author>
<name>Ioannis Aslanidis</name>
<email>deathwing00@deathwing00.org</email>
</author>
<published>2009-11-25T18:02:13Z</published>
<link rel='alternate' type='text/html' href='http://labs.feurix.org/admin/pylsyncd/commit/?id=8b9fd2afdc2a47b2ae007f9117f6ba554d1eaa39'/>
<id>8b9fd2afdc2a47b2ae007f9117f6ba554d1eaa39</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Update development URL</title>
<updated>2009-11-25T17:02:20Z</updated>
<author>
<name>John Feuerstein</name>
<email>john@feurix.com</email>
</author>
<published>2009-11-25T17:02:20Z</published>
<link rel='alternate' type='text/html' href='http://labs.feurix.org/admin/pylsyncd/commit/?id=852429e82d5b92bf49e7aeb56276bc08f1f57c7b'/>
<id>852429e82d5b92bf49e7aeb56276bc08f1f57c7b</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Handle inotify IN_Q_OVERFLOW</title>
<updated>2009-11-11T03:48:47Z</updated>
<author>
<name>John Feuerstein</name>
<email>john@feurix.com</email>
</author>
<published>2009-11-11T03:48:47Z</published>
<link rel='alternate' type='text/html' href='http://labs.feurix.org/admin/pylsyncd/commit/?id=181006144d78cd94bdc0516f5a433b7dab781b8e'/>
<id>181006144d78cd94bdc0516f5a433b7dab781b8e</id>
<content type='text'>
Handle inotify event queue overflows (raising IN_Q_OVERFLOW) by adding a
full recursive sync to the queue. This can happen if you have a very
active source and overstep the max_queued_events inotify variable.
(You may increase it using sysctl -- needs super user privileges)
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Handle inotify event queue overflows (raising IN_Q_OVERFLOW) by adding a
full recursive sync to the queue. This can happen if you have a very
active source and overstep the max_queued_events inotify variable.
(You may increase it using sysctl -- needs super user privileges)
</pre>
</div>
</content>
</entry>
<entry>
<title>Fix log message wording</title>
<updated>2009-11-11T02:28:29Z</updated>
<author>
<name>John Feuerstein</name>
<email>john@feurix.com</email>
</author>
<published>2009-11-11T02:28:29Z</published>
<link rel='alternate' type='text/html' href='http://labs.feurix.org/admin/pylsyncd/commit/?id=95ef3d0084b12d2d24502a3de417dc92cbf000db'/>
<id>95ef3d0084b12d2d24502a3de417dc92cbf000db</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Implement support for rsync file-lists to reduce forking</title>
<updated>2009-11-10T22:40:06Z</updated>
<author>
<name>John Feuerstein</name>
<email>john@feurix.com</email>
</author>
<published>2009-11-10T22:40:06Z</published>
<link rel='alternate' type='text/html' href='http://labs.feurix.org/admin/pylsyncd/commit/?id=18124238bf7c9496a22c11553efff18b248238d5'/>
<id>18124238bf7c9496a22c11553efff18b248238d5</id>
<content type='text'>
This change is a *HUGE* performance improvement. Instead of forking one
rsync process for each and every changed source path, we assemble a
rsync-compatible file-list and hand it over to rsync on stdin.

In addition we don't need to assemble source paths containing the
virtual root marker anymore. We let rsync handle this, based on the fact
that all paths within a file-list are relative to the source given on
the command line.

Since we want to optimize the synchronization and need to distinguish
between recursive and non-recursive rsync invocations, the final changes
are:

    1. ItemQueue now has attributes called "dirs" and "trees".

        These are two lists to hold non-recursive dir paths
        ("dirs") and recursive dir paths ("trees"). We prevent
        duplicates and try to optimize out as much as possible.

    2. After ItemQueue optimization the rsync file-lists are assembled

        These rsync file-lists are \0 delimited to support all possible
        pathnames. Also, all paths in the file-list are relative to the
        given rsync source path. The old virtual root ("target path
        rewriting") behaviour is preserved, ie. the full path is
        preserved on the target (default) -- or, if you specify a
        virtual root within the source path, all target paths will be
        relative to this virtual root.

    3. At most two rsync invocations per cycle are done now

        One running recursively with all "trees" and the other running
        non-recursively with all "dirs". If one of both variants has an
        empty queue, we silently skip it.

The old "failsafe" behaviour is preserved, ie. if a destination fails to
sync, the queued items won't be dropped and will be handled by the next
run... (until we hit a limit and remove the destination, as always).
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This change is a *HUGE* performance improvement. Instead of forking one
rsync process for each and every changed source path, we assemble a
rsync-compatible file-list and hand it over to rsync on stdin.

In addition we don't need to assemble source paths containing the
virtual root marker anymore. We let rsync handle this, based on the fact
that all paths within a file-list are relative to the source given on
the command line.

Since we want to optimize the synchronization and need to distinguish
between recursive and non-recursive rsync invocations, the final changes
are:

    1. ItemQueue now has attributes called "dirs" and "trees".

        These are two lists to hold non-recursive dir paths
        ("dirs") and recursive dir paths ("trees"). We prevent
        duplicates and try to optimize out as much as possible.

    2. After ItemQueue optimization the rsync file-lists are assembled

        These rsync file-lists are \0 delimited to support all possible
        pathnames. Also, all paths in the file-list are relative to the
        given rsync source path. The old virtual root ("target path
        rewriting") behaviour is preserved, ie. the full path is
        preserved on the target (default) -- or, if you specify a
        virtual root within the source path, all target paths will be
        relative to this virtual root.

    3. At most two rsync invocations per cycle are done now

        One running recursively with all "trees" and the other running
        non-recursively with all "dirs". If one of both variants has an
        empty queue, we silently skip it.

The old "failsafe" behaviour is preserved, ie. if a destination fails to
sync, the queued items won't be dropped and will be handled by the next
run... (until we hit a limit and remove the destination, as always).
</pre>
</div>
</content>
</entry>
<entry>
<title>Do not follow symlinks when creating watches</title>
<updated>2009-11-10T22:36:47Z</updated>
<author>
<name>John Feuerstein</name>
<email>john@feurix.com</email>
</author>
<published>2009-11-10T22:36:47Z</published>
<link rel='alternate' type='text/html' href='http://labs.feurix.org/admin/pylsyncd/commit/?id=c88064ab314b53c71cbcd93adafbfb1b1f0db803'/>
<id>c88064ab314b53c71cbcd93adafbfb1b1f0db803</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
</feed>

