Par2 thread hangs checking blocks (causes other things to fail)

[thebeing]

The following bug has happened in all versions of Alt.Binz I've used (Free version & 3.7.1 - 3.8.5). It has finally gotten annoying enough to report.

Occasionally the Par2 thread will hang and stop adding completed blocks to the Par2 tab. This also causes it to not add any future Par2 files downloaded to the Par2 tab and also fail to join/extract anything downloaded after that point. The problem has a tendency to occur where there are multiple files and Par2 joined into a single collection on the Download Queue tab. I've yet to see it occur when each file has its own entry on the Download Queue, so possibly that would be were to start looking for the bug. This most recent time it happened with a 25GB split file in a 58GB collection, but that is an extreme. The hang may happen more often with massive files, but I believe I've seen it happen with smaller files as well.

This will also cause files to fail to join, extract, and/or repair (including Alt.Binz automatically unpausing and downloading Par2 files needed for repair). In the last few revisions Alt.Binz now even fails to save the Par2 tab on exit when this hang has occured. I've looked at the log and see nothing interesting. Even though detail logging was checked, I see no mention of Par2 activity anywhere in the log.

This is all on WinXP SP3 x86. Other than what I've mentioned, I'm unsure how to reliably reproduce the bug, as it seems to occur at random.

[thebeing]

As a follow-up of this, I reproduced it again, this time with ~175 split files (~10 parts each) in a single AltBinz collection on the Download Queue. After the 25th download had completed I checked Altbinz and the par2 thread malfunctioned again. On the par2 tab, only the first 10 downloads were shown as 100% blocks, 11-15 were shown with ~0.5% blocks, and no par2 for 16-25 were listed at all.

I paused the downloads, and shortly after AltBinz jumped to 100% CPU and appears to be attempting to recover, but something is still broken. It's been an hour paused now, and it's only joined 4 new files, and added added all the blocks for 3 files to the par2 tab. I'll leave it alone for more hours and see it is able to fully recover. I dread what is going to happen for the remaining 150 files... I may literally need to leave Altbinz eating 100% CPU for days after everything finishes downloading... In the past I've also seen that removing par2 files from the par2 will also hang when Altbiz is in this bugged state.

Rdl, I hope you are able to track down the cause. PAR2 bug? Threading bug? Something else? The computer I'm seeing the bug on has an AMD X2 4800+ (939) dual-core, if that matters. I'm convinced the bug is triggered when downloading many files in a single collection. Unfortunately, it also may be easier to reproduce than I originally thought.

Update1: It did eventually join and add the blocks from the completed files to the par2 tab, but the bug remained when I resumed downloading. No new par2 files were added while downloading. Pausing still resulted in Altbinz using 100% CPU, slowly reading files for a long time, attempting to recover.

Update2: The bug remained after exiting and re-opening Altbiz and resuming downloading. This suggests Altbinz has bugged the entire collection. The par2 files didn't disappear either, possibly since I let Altbinz do its thing until it was idle.

Update3: After further investigation with Process Monitor, it looks like Altbinz is getting stuck in a loop when parsing files. It is doing ~30 loops reading each file part in 640KB chunks (which happens to be the par2 block size) at 1280KB offsets for a total of ~1500 read operations per file part in this case. This may explain the the cause of 100% CPU usage, and why this thread continues to fall way behind the other threads. Now you just need to find out why this abnormal behavior is happening. When this bug isn't happening, I am able to download things in Altbinz with downloading, par2 checking, par2 repair, joining, extracting all happening in succession.

Update4: I tried splitting the uncompleted part of the collection into a new collection, exiting and re-opening Altbinz and resuming download, but I still have the bug.

Update5: This time I tried exporting the collection to an NZB, deleting everything from the Download Queue, and re-adding the NZB. Still bugged... This definitely does seem like a par2 bug.  As I mentioned previously these are multiple split files within a single collection, 25MB per part, yENC, and the par2 files being reported by Altbinz as 640000 bytes block size & created by par2cmdline 0.4.

[thebeing]

Update6: I rebooted, opened Altbinz, downloaded the next file, and paused. The problem got worse somehow. Now the par2 thread seems to be doing ~110 loops for each part in 640KB chunks (~4000 read operations @~11MB/s). At the rate it's going, it looks like it will take around an hour for Altbinz to add all the blocks to the par2 tab and join the split parts for just this single 250MB file...

Update7: I tried splitting a single file's split part into it's own collection, still bugged. Whatever this is, it's carrying over when splitting this collection in the Download Queue, no matter how big or small the split.

[Hecks]

Clearly it's an issue with a particular post, so you'll need to upload the NZB somehere and link it here so it can be checked out. Try to isolate the problem to just one split file.

[thebeing]

Update8:By removing all the par2 files from the par2 tab, deleting the next par2 file from the download directory, and re-downloading an NZB (which contains the par2) for just a next file, I was able to workaround the bug. Blocks were added to the par2 and the file was joined as parts were downloaded.

Update9:For the time being, I guess I'll just download the massive collections without any par2 in the par2 tab. Adding the par2 files one by one, seems to add blocks to the par2 tab and join the split files promptly, avoiding the bug.

[thebeing]

Clearly it's an issue with a particular post, so you'll need to upload the NZB somehere and link it here so it can be checked out. Try to isolate the problem to just one split file.

Do you still think this is the case, taking into account Update 8 & 9? Adding the previously bugged exported NZB, without any par2 in the par2 tab works fine. This seems to confirm it's just a bug with par2 handling. Doesn't it?

[Hecks]

Par2 works fine in 99% of cases, and there's no way of testing your case without actually trying the download, so an NZB is needed.

Edit: sorry I'm losing track of your updates. It seems that the clearest solution is not to dump everything in one collection (there's really no need to), and make sure each collection is downloaded to its own folder. Put together a test case NZB if you want a better solution for handling multiple par2s in the same folder.

[thebeing]

I sent you a PM with the NZB created on binsearch which triggered the bug this time.

I do think it is related to some sort of behavior (like multiple par2 files in a folder as you mentioned), rather than a buggy NZB. What triggered the bug in my first post were 12 singular NZBs (of 12 DVD9 ISO files) from a different source, which I joined into a single collection with Altbinz.

When the bug is triggered, the solution appears to be removing all the par2 files from the par2 tab, deleting par2 in your download dir for files not yet downloaded, and re-downloading everything from scratch along with par2.

[Hecks]

I've prepared the sacrificial virgins and summoned the Kraken to look at your NZB, so let's see.

I doubt very much that you'll see this prob again if you apply a bit of discipline to using Alt.Binz 'cleanly', i.e. always keep one distinct posting in each collection (archive + pars, etc), and always download each collection to a separate folder (Setup > Download > 'Create subfolders based on NZB name') to keep things properly isolated.

[thebeing]

I doubt very much that you'll see this prob again if you apply a bit of discipline to using Alt.Binz 'cleanly', i.e. always keep one distinct posting in each collection (archive + pars, etc), and always download each collection to a separate folder (Setup > Download > 'Create subfolders based on NZB name') to keep things properly isolated.

Even if that's the case, it would really be nice to see this bug fixed (I've run into it at least 20% of the time for the minimal things I download from usenet). Keeping everything isolated into its own folder means time spent manually moving files around and deleting folders, which has gotten annoying over time, and why I try to avoid doing so when possible... It's so much nicer to just download things which belong in the same folder, directly to said folder.

The bug can't just be multiple par2 files in the download dir, since I still had a lot of par2 in that folder, and dragging them in one by one works. Does Altbinz check par2 files sequentially or in parallel? Does Altbinz do anything when a new par2 file is downloaded and added to the par2 tab? If adding a new par2 to the par2 tab causes in-progress block checking to be interrupted in favor of the new par2, I could see how that could eventually cause things to cascade over time.

As for the endless par2 read loops I saw, it may suggest one of two things. par2 is working correctly, but Altbinz isn't listening when par2 reports the blocks it checked. Or par2 continues to loop because Altbinz is requesting block information and par2 isn't giving it the first time around. Since Altbinz used a modded par2 exe, I wonder if the looping behavior on purpose (aka a retry behavior) when par2 and Altbinz aren't communicating properly?

In any case, I'll try to figure something out. I wonder if just just making the par2 files the last things downloaded would workaround the issue.

I hope someone can reproduce, and this isn't some bug with my CPU or something. These AMD X2 (939) CPUs did have that processor core synchronization bug. Though using the /usepmtimer switch with the updated processor driver as I'm currently using should avoid the issue. Yet if this is indeed a threading bug, rather than a par2 bug, and AltBinz shares very tight timing data between cores/threads, I wonder if Altbinz requires the Dual-Core Optimizer driver. That Dual-Core Opt driver is basically a hack that makes an extra effort to guarantee the cores are always in sync sharing the same global clock, but being a hack, it conflicts with certain programs I use. If Altbinz is one of the rare applications which has the potential to suffer from this processor bug, maybe this would give Rdl an idea of what to fix. Doubtful that this is that issue, but worth mentioning for those not aware. I'll eventually get around to testing this possibility, but I don't see it as a viable long-term solution in my case.

What triggered the bug in my first post were 12 singular NZBs (of 12 DVD9 ISO files) from a different source, which I joined into a single collection with Altbinz.

Oops, I got confused, that was something else. The first post that got bugged were 16 singular NZBs for MKVs (7 were 20-70MB, 6 were 2GB-3GB, 1 was 6GB, 1 was 9GB, 1 was 23GB) joined into a single collection with Altbinz.

[Hecks]

Well by contrast I would say that I have problems with par2 about once a month (usually a problem with the par2 files themselves). To repeat: there's no benefit at all with merging NZBs like you're doing, use the queue as it's designed to be used.

The standard setup for Alt.Binz is as I've described. Download to separate subfolders on one drive/partition, unrar/join to another, have Alt.Binz automatically remove your empty dirs, use article caching in RAM and be a happy bunny. There's no need to do anything manually, Alt.Binz is designed from the bottom up to be automated.

As for the rest, let's wait and see.

[thebeing]

About that NZB I PM'ed you, I sorted it alphabetically in Altbinz since that NZB had everything in reverse order. This ended up with the par2 files getting placed after their respective split parts. This isn't particularly noteworthy, since the bug has also happened with par2 files in their normal location (before respective split parts), but if you want everything exactly the same...

I've yet to really fit unrar/join directory into my usage of Altbinz. It still takes some manual work to specify a unique unrar/join folder for a collection, doesn't it? It's not fun to specify folders one by one. I assume it's not currently possible to select multiple collections in the download queue, and set their download and/or unrar folder to the same unique directory, all at once?  Or is it?

Unfortunately having everything dumped in a global directory for sorting, doesn't really work well with my current drive/volume organization & free-space strategy.

[thebeing]

As I mentioned previously, I decided to just download the huge collection without NZBs already in the par2 tab, and just drag in par2 in batches for completed files. Well I just ran into a slightly different, but likely related, variation of the bug.

I dragged in ~15-30 par2 files so Altbinz would join those completed downloads. The first couple went fine, but then one of them only added blocks and joined the first split part without finishing, and then skipped to the next par2. Altbinz only checking blocks for 0-2 parts before skipping the par2 for the next, seems to be a common issue with this bug. If this is related to the cause of the core issue, it may be easier to reproduce the bug by just moving groups of completed files w/ par2 in a directory and dragging in groups of par2 files into Altbinz for checking, over and over again, while other things are downloading.

[Hecks]

At this stage, you're really adding no new information at all. Who knows what you're piling into the same directory and what the potential collisions might be there.

Continue your unwise decision not to split your large collection into smaller ones and dl to subfolders, by all means, but there's really no need to keep posting about it.

[thebeing]

Who knows what you're piling into the same directory and what the potential collisions might be there.

Continue your unwise decision not to split your large collection into smaller ones and dl to subfolders, by all means, but there's really no need to keep posting about it.

To answer this question of what I'm piling into the same directory, truthfully not much.  I do always place each collection into it's own unique empty directory. The only potential collisions would be when I join multiple collections (usually of 2-15 files) with related items into a single collection and unique empty directory. In case I wasn't clear, this bug happens with very small collections as well. I've seen it occur after only two to three 250MB files (out of around five in a collection) were downloaded to the same empty directory. What this basically means, is I can never join 2 or more NZB collections (downloaded to an empty directory) or I risk experiencing this par2 bug. At least that is how it seems.

I do understand what you're saying, and will try out some of things you suggested so I can continue using Altbinz effectively. At this point I have indeed said all that can be said about this bug, that it really does exist, what may trigger it, and how much I would like to see it fixed. I apologize if all the posts were a bit much. I was only trying to document as much information as possible, no matter how trivial, since for all I know some little piece of info could become useful in tracking down and fixing the issue. Over the years I've done a lot of beta testing for various applications, so I've seen time and time again how difficult issues such as these can be to track down and fix, especially if the developer is unable to reproduce the issue themselves. That this bug has to some extent existed in every version of Altbinz, and if I'm the first to actively complain about it, it makes chances of it getting fixed all the less likely. I can only hope Rdl gets insanely lucky and has a break-through with an idea of how to fix it.

  Once again, sorry for all the trivial and semi-repetitive posts. I'll restrain myself and post no more in this thread, unless my feedback is required.