Posts: 2
Threads: 1
Joined: Aug 2015
Posts: 2
Threads: 1
Joined: Aug 2015
Interesting problem I'm having: LRTimelapse takes an incredibly long time to load my main folder of timelapses. It's inconsistent how long it'll take, but it's usually between about 15 minutes and several hours. I've been having this problem for well over a year, and it seems to persist with every update of LRT. I also have LRT on my Mac Pro and on my MacBook Pro, and it does this on both machines.
So, a little bit of background on my setup. All my timelapse sequences (RAW frames) are on a Synology NAS drive accessed over a gigabit ethernet network. The volume size is about 40TB. Speed-wise, the bottleneck is the local network, but I get better speeds off this than the 1TB Firewire 800 drives I tend to travel with. I have no doubt this is the underlying problem, but there just isn't really a good workaround. Once LRT loads the initial directory tree, it's just as fast as working off a local drive, and I have the benefit of keeping things organized, protection against drive failures, and my automatic backup/syncing system which goes to a second mirror. The time spent moving 20-50GB sequences to a local drive and back is going to create more problems than it'd solve... and this way I can also keep my AE render queue running on other machines which access the same NAS.
As I navigate through the file tree, the first few directories load fine, but as soon as I hit the little arrow to open the "Scenes" folder, LRT just hangs, or at least appears to. I can't scroll, I can't click on any menus on LRT, it's just dead for a seemingly arbitrary amount of time until that list loads, and then everything works fine. During this period, CPU usage is minimal, network activity is minimal, but clearly something is happening in the background because the disk access lights on the Synology are going nuts. Here's the interesting thing: I also can't access the Synology from that Mac. Any attempt to get a directory listing will beachball the Finder for an excessively long time. I can, however, access the Synology from other machines on the network, and access speeds are normal.
A few other notes:
- The number of files in the "Scenes" directory seems to make load times exponentially longer. I've got to the point where I've started splitting my "Scenes" folder into separate directories for motion-controlled scenes, scenes I've shot in Europe, that kind of thing, but this sorta defeats the purpose of having everything in one master directory with consistent paths for ease of distributed rendering.
- Typically if I haven't opened LRT yet, I'll wait until I'm about to leave for awhile or before I go to bed, and it'll just be ready to go when I get home. Then, I just leave LRT open and never turn the computer off. My computer crashed this morning (unrelated) so I spent a good chunk of the afternoon waiting for LRT to open so I could work (in fact, I'm still waiting), so I was inspired to make this post. It's gone from a minor inconvenience to actually being a significant setback in my workflow.
- LRT is pretty good about remembering where that folder is on the NAS, so if I just open LRT and walk away for awhile, it's usually there when I get back. Except when I forget to connect to the fileserver first, in which case LRT will default to the filesystem root, and everything takes 3x as long.
- For what it's worth, Lightroom lags on directory listings in much the same way, but Lightroom's lag is usually only several minutes (and probably because it already knows what files and folders to look for instead of building that list from scratch each time)
I don't know what the solution is, and I really hope it's not "stop accessing everything from a RAID fileserver over your network, dummy", because that's an integral part of my workflow. I have a feeling it has to do with the method that LRT is using to get the counts of files in the respective folders... it is literally counting over a million RAW files in about a thousand subdirectories (one subdirectory for every timelapse I've ever shot) every time I load this main directory.
I'm certainly open to suggestions, it's entirely possible that I'm just doing something wrong, but I feel like I've tried everything while keeping my workflow pretty consistent. Still, maybe there is a solution... timelapses generate so much data that the only workable solution if you're shooting lots of them and want to keep them consistently organized is to keep them on a RAID, NAS, or other kind of fileserver that can handle volumes larger than it's individual component disks. The Synology is an incredibly popular device for this kind of thing so I'd be very surprised if there aren't other LRT users keeping their master RAW frames on such a system. I've got an 8 drive system crammed with 6TB drives and it's *all* just RAW timelapse frames.
That considered, I'd also like to propose some future feature requests:
- Directory listing caching of some kind, like how Lightroom works. Index the directory listing once, and store it somewhere so LRTimelapse can access those directories when clicked. If the fileserver is disconnected when a folder is clicked on to load the directory, then generate an error. There's already a "reload" button for refreshing the directory listing, so it's already basically being cached somehow, but I'd love it if it was saved or cached between LRT launches.
- Ability to turn off file counting within the directory listing. I'm pretty sure this is the root of the problem as it's effectively crawling millions of files over a network. Even with network latency being as low as it is, I'll bet the milliseconds are adding up to hours here.
Anyway, thanks for reading, I really appreciate the effort put into this software. It's become central to my ability to shoot and edit timelapse sequences.
TLDR version: I'm using LRTimelapse to load massive directories from a RAID network fileserver and it is SLOW, although once the directory is loaded on the left column everything is nice and fast.
-Hal Bergman
halbergman.com
So, a little bit of background on my setup. All my timelapse sequences (RAW frames) are on a Synology NAS drive accessed over a gigabit ethernet network. The volume size is about 40TB. Speed-wise, the bottleneck is the local network, but I get better speeds off this than the 1TB Firewire 800 drives I tend to travel with. I have no doubt this is the underlying problem, but there just isn't really a good workaround. Once LRT loads the initial directory tree, it's just as fast as working off a local drive, and I have the benefit of keeping things organized, protection against drive failures, and my automatic backup/syncing system which goes to a second mirror. The time spent moving 20-50GB sequences to a local drive and back is going to create more problems than it'd solve... and this way I can also keep my AE render queue running on other machines which access the same NAS.
As I navigate through the file tree, the first few directories load fine, but as soon as I hit the little arrow to open the "Scenes" folder, LRT just hangs, or at least appears to. I can't scroll, I can't click on any menus on LRT, it's just dead for a seemingly arbitrary amount of time until that list loads, and then everything works fine. During this period, CPU usage is minimal, network activity is minimal, but clearly something is happening in the background because the disk access lights on the Synology are going nuts. Here's the interesting thing: I also can't access the Synology from that Mac. Any attempt to get a directory listing will beachball the Finder for an excessively long time. I can, however, access the Synology from other machines on the network, and access speeds are normal.
A few other notes:
- The number of files in the "Scenes" directory seems to make load times exponentially longer. I've got to the point where I've started splitting my "Scenes" folder into separate directories for motion-controlled scenes, scenes I've shot in Europe, that kind of thing, but this sorta defeats the purpose of having everything in one master directory with consistent paths for ease of distributed rendering.
- Typically if I haven't opened LRT yet, I'll wait until I'm about to leave for awhile or before I go to bed, and it'll just be ready to go when I get home. Then, I just leave LRT open and never turn the computer off. My computer crashed this morning (unrelated) so I spent a good chunk of the afternoon waiting for LRT to open so I could work (in fact, I'm still waiting), so I was inspired to make this post. It's gone from a minor inconvenience to actually being a significant setback in my workflow.
- LRT is pretty good about remembering where that folder is on the NAS, so if I just open LRT and walk away for awhile, it's usually there when I get back. Except when I forget to connect to the fileserver first, in which case LRT will default to the filesystem root, and everything takes 3x as long.
- For what it's worth, Lightroom lags on directory listings in much the same way, but Lightroom's lag is usually only several minutes (and probably because it already knows what files and folders to look for instead of building that list from scratch each time)
I don't know what the solution is, and I really hope it's not "stop accessing everything from a RAID fileserver over your network, dummy", because that's an integral part of my workflow. I have a feeling it has to do with the method that LRT is using to get the counts of files in the respective folders... it is literally counting over a million RAW files in about a thousand subdirectories (one subdirectory for every timelapse I've ever shot) every time I load this main directory.
I'm certainly open to suggestions, it's entirely possible that I'm just doing something wrong, but I feel like I've tried everything while keeping my workflow pretty consistent. Still, maybe there is a solution... timelapses generate so much data that the only workable solution if you're shooting lots of them and want to keep them consistently organized is to keep them on a RAID, NAS, or other kind of fileserver that can handle volumes larger than it's individual component disks. The Synology is an incredibly popular device for this kind of thing so I'd be very surprised if there aren't other LRT users keeping their master RAW frames on such a system. I've got an 8 drive system crammed with 6TB drives and it's *all* just RAW timelapse frames.
That considered, I'd also like to propose some future feature requests:
- Directory listing caching of some kind, like how Lightroom works. Index the directory listing once, and store it somewhere so LRTimelapse can access those directories when clicked. If the fileserver is disconnected when a folder is clicked on to load the directory, then generate an error. There's already a "reload" button for refreshing the directory listing, so it's already basically being cached somehow, but I'd love it if it was saved or cached between LRT launches.
- Ability to turn off file counting within the directory listing. I'm pretty sure this is the root of the problem as it's effectively crawling millions of files over a network. Even with network latency being as low as it is, I'll bet the milliseconds are adding up to hours here.
Anyway, thanks for reading, I really appreciate the effort put into this software. It's become central to my ability to shoot and edit timelapse sequences.
TLDR version: I'm using LRTimelapse to load massive directories from a RAID network fileserver and it is SLOW, although once the directory is loaded on the left column everything is nice and fast.
-Hal Bergman
halbergman.com
