[Bug 289330] New: Album file watch failing (QKqueueFileSystemWatcherEngine:addPaths: open: Too many open files in system)

[brad]

https://bugs.kde.org/show_bug.cgi?id=289330

           Summary: Album file watch failing
                    (QKqueueFileSystemWatcherEngine:addPaths: open: Too
                    many open files in system)
           Product: digikam
           Version: 2.4.0
          Platform: Compiled Sources
        OS/Version: OS X
            Status: UNCONFIRMED
          Severity: normal
          Priority: NOR
         Component: general
        AssignedTo: [hidden email]
        ReportedBy: [hidden email]


Version:           2.4.0 (using KDE 4.7.3)
OS:                OS X

It appears that since the commit :
https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/dcf2ba9d7e92d51b049bdd354f82ddbf75e51a31

digikam does not load images in OS X. The console error that i receive
(thousands of times) is the following:

  QKqueueFileSystemWatcherEngine::addPaths: open: Too many open files in system
  QFileSystemWatcher: failed to add paths:
/Volumes/Home/Directory/Pictures/Savedpictures/Pic0193.jpg
  QKqueueFileSystemWatcherEngine::addPaths: open: Too many open files in system
  QFileSystemWatcher: failed to add paths:
/Volumes/HomeDirectory/Pictures/Savedpictures/Pic0194.jpg


This might be related to the following 'fixed' QT bug:
https://bugreports.qt.nokia.com//browse/QTBUG-2478


Reproducible: Always

Steps to Reproduce:
compile and run digikam from the latest git, then try to load an image.

Actual Results:  
image will not load. GUI will report failed to load image.

Expected Results:  
To see image loaded.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=289330


Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]
          Component|general                     |Database




--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[brad]

https://bugs.kde.org/show_bug.cgi?id=289330





--- Comment #1 from brad <bkn ithryn net>  2011-12-24 20:45:54 ---
Created an attachment (id=67089)
 --> (http://bugs.kde.org/attachment.cgi?id=67089)
small patch to limit 'watched' files on OS X.

I propose the attached patch to fix this problem on OS X.  OS X limits the
number files opened per process to be 255, so if the user has more than 255
files, they cannot open them, which doesn't make digikam all that useful. This
patch makes digikam useful again but only watching directories, this limits the
user to 255 directories of images. This is not a perfect solution, but for the
timebeing this will enable digikam to be usable on OS X. I do not think this
bug should be closed since this is just a work-around and not a genuine fix.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=289330


Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Album file watch failing    |Album file watch failing
                   |(QKqueueFileSystemWatcherEn |(QKqueueFileSystemWatcherEn
                   |gine:addPaths: open: Too    |gine:addPaths: open: Too
                   |many open files in system)  |many open files in system)
                   |                            |[patch]




--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=289330





--- Comment #2 from Gilles Caulier <caulier gilles gmail com>  2011-12-24 21:55:21 ---
Marcel,

Fine for you to apply this patch. It sound like not too intrusive and without
any action under non OSX system...

Gilles Caulier

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=289330





--- Comment #3 from Marcel Wiesweg <marcel wiesweg gmx de>  2011-12-26 21:32:42 ---
Apparently the underlying Qt bug is not fixed? Are you running Qt 4.7 and Mac
OS X >= 10.5?
Even with the number limited to 255, digikam would not be able to open more
than one other file. I'd rather disable directory watching if the FSEvents API
is not available

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[brad]

https://bugs.kde.org/show_bug.cgi?id=289330





--- Comment #4 from brad <bkn ithryn net>  2011-12-27 01:00:02 ---
On Dec 26, 2011, at 4:32 PM, Marcel Wiesweg wrote:

> https://bugs.kde.org/show_bug.cgi?id=289330
>
>
>
>
>
> --- Comment #3 from Marcel Wiesweg <marcel wiesweg gmx de>  2011-12-26 21:32:42 ---
> Apparently the underlying Qt bug is not fixed? Are you running Qt 4.7 and Mac
> OS X >= 10.5?

Yes i'm running current QT 4.7.4, and OS X 10.6.8. I have a simple test program
i can post if you have access to an OS X machine and want to look at.

the qt bug is not fixed. it really seems to be a bug (short fall) of OS X not
QT. eitherway, a process can't open more than 255 files without some system
manipulation which i wouldn't expect end users to do.

> Even with the number limited to 255, digikam would not be able to open more
> than one other file. I'd rather disable directory watching if the FSEvents API
> is not available
>
If you'd rather disable file watching, then go for it. I'm not familiar with
the file watching code and i wanted to make the smallest change possible to the
code to get it running on OS X.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=289330





--- Comment #5 from Marcel Wiesweg <marcel wiesweg gmx de>  2011-12-28 10:26:08 ---
According to API docs, Qt would be using FSEvents on Mac OS X >= 10.5:
http://en.wikipedia.org/wiki/FSEvents

http://developer.qt.nokia.com/doc/qt-4.8/qfilesystemwatcher.html
"Mac OS X 10.5 and up use a different backend and do not suffer from this
issue."

Now looking into the source code of Qt 4.7.3, apparently the creation of the
FSEvents backend is commented out
(QFileSystemWatcherPrivate::createNativeEngine()). This is still true on Qt
master on gitorious. So apparently the API docs are wrong here - the backend
does not seem to be used. I would recommend to report this bug to Qt, or try to
reopen the bug report mentioned above.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[brad]

https://bugs.kde.org/show_bug.cgi?id=289330





--- Comment #6 from brad <bkn ithryn net>  2011-12-29 22:50:55 ---
(currently i'm compiling qt with fsevents uncommented to see if it will work)

after searching around the net, it appears that QT has gone back and forth from
using fvevents to kqueue. it's not clearly to me why, but i think its because
of bugs in fsevents on apple's side.

In the meantime it seems that disabling file waching (or album watching) on OS
X is right way to go for digikam until this QT/OS X bug is resolved.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[brad]

https://bugs.kde.org/show_bug.cgi?id=289330


brad <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #67089|0                           |1
        is obsolete|                            |




--- Comment #7 from brad <bkn ithryn net>  2012-01-11 15:19:09 ---
Created an attachment (id=67697)
 --> (http://bugs.kde.org/attachment.cgi?id=67697)
simple patch that should apply smoothly to current source.

Trying to get better at git, creating patch with git format-patch. To apply use
git am 0001-disable-filewatching-on-OS-X-but-dir-watching-still-.patch in the
digikam-sc/core directory.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[brad]

https://bugs.kde.org/show_bug.cgi?id=289330





--- Comment #8 from brad <bkn ithryn net>  2012-01-29 23:40:13 ---
Here is an update (blog post) on qfilesystemwatcher for QT5. In the comments
the author can get over 10k watched files on OS X. I'm try to contact the
author to get more info. haven't heard anything yet.

http://blog.rburchell.com/2012/01/qfilesystemwatcher-internals-in-qt-5.html

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=289330


[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]




--- Comment #9 from  <g2florette gmail com>  2012-02-21 23:11:47 ---
Hi,

Compiling digikam 2.6 from git on MacOSX Lion. I've the "too many open files"
problem with my > 10K images collection : digikam isn't able to start. Without
any collection, it runs fine...
The patch from Brad, applied like showed in comment #7, fixes the problem : I
can use digikam with my whole collection without any problem.
Hope this small report helps...

G.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=289330

--- Comment #10 from Jan Gosmann <[hidden email]> ---
With Digikam 2.5.0 and Mac OS X Lion I had also the "too many open files"
problem leading to a crash of digikam at startup. With the patch from comment
#7 I can start Digikam.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=289330

--- Comment #11 from Jan Gosmann <[hidden email]> ---
*** Bug 292522 has been marked as a duplicate of this bug. ***

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=289330

--- Comment #12 from Marcel Wiesweg <[hidden email]> ---
I will apply the patch by brad. Then I consider this bug fixed; it is not, but
the real fix is beyond our scope, in OS X and Qt.
Any better solution is welcome any time.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=289330

Marcel Wiesweg <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |FIXED

--- Comment #13 from Marcel Wiesweg <[hidden email]> ---
Git commit 5e6b607b38076eb226a7fa92c82f0fe476bc9681 by Marcel Wiesweg.
Committed on 14/04/2012 at 17:12.
Pushed by mwiesweg into branch 'master'.

Apply patch by Brad <[hidden email]> to disable file watch on OS X
until this is properly supported by this platform.

M  +2    -1    NEWS
M  +7    -0    digikam/album/albumwatch.cpp

http://commits.kde.org/digikam/5e6b607b38076eb226a7fa92c82f0fe476bc9681

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=289330

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Version Fixed In|                            |2.6.0

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=289330

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Platform|Compiled Sources            |MacPorts Packages

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=289330

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|Database                    |Database-Scan

--
You are receiving this mail because:
You are the assignee for the bug.