Direct Write of downloads

[wizziwig]

I'm always interested in finding ways to make altbinz the best nzb downloader available.

I noticed that altbinz writes about double the total data to my disk compared to nzbget.  For example, when downloading a 4GB file, altbinz will write 8GB total to the download folder disk.  Nzbget will write ~4GB - same as the original file size.

By looking at resource monitor, I can see the reason.  Altbinz writes all parts to small temporary files inside the temp folder.  It then copies these small parts again into the final large file.  Nzbget writes each part directly into the final .rar file at the correct file offset.  The OS automatically pads the missing space when you seek beyond the end of the file so there are no redundant writes or holes.  Once the entire files is fully filled in, it then "moves" the file into the download folder.  This MoveFile() function is instant because it just updates the directory structure - no data is copied again.  You end up with half as many bytes written to disk before you're ready to unrar.

This feature could speed up downloading and reduce wear on SSD.  I think you would need to cache the articles in memory so you can decode them into the right position of the full file.

Most other nzb downloaders operate similar to altbinz.  I think only nzbget and maybe newsleecher (will have to verify) use direct writes of articles.

[Rdl]

There is already such option. Setup->Download #2-> Article caching
Certainly wont work effectively for 4GB file.

[wizziwig]

That's what I expected but it's still creating those small files inside the temp folder.  This was with an article cache of 1GB and download of a ~40 GB file.

I think what might be happening is that the decoding is much slower and lagging behind my downloading (450 Mbit).  For example, I can see in the queue that I have already downloaded 200 files but it's still decoding file 60.  So the 1GB cache is not enough for all the undecoded articles.  The only fix would be to pause downloading until the decoder catches up or to speed up the decoding.

Can we make the decoding multi-threaded so it can decode multiple articles at once?  I have lots of unused cores on my CPU.  Somehow nzbget can download and decode at full speed with a similar 1GB RAM cache without using any extra disk cache.  I didn't check how many threads they are using.  Maybe their decoder is just faster even with a single thread.

By the way, the current 0.4xx altbinz is already much faster than the older versions.  You've done a very good job optimizing it.  I'm just trying to find some ideas to make it even better.

[wizziwig]

I verified that the decoding was the problem.  Was downloading something that was split into a few hundred 100 MByte rar files.  By the time it downloaded 38 rar files, it was still waiting to decode 24 of the previous ones.  When the decoder lag was less than 10 files, the 1GByte article cache was working perfectly and it wasn't using my disk cache in temp folder at all.

I guess the current decoding is just too slow if you have a very fast internet connection.  My SSD is fast enough at 400 Mbyte/sec write speed (task manager shows max 20% usage).  But my CPU single thread maximum is 3.5 Ghz which is not fast enough to keep up with downloads.

Most rar files are broken down into multiple articles.  We would need to either decode multiple articles or multiple rar files at the same time.  Have to be careful because too much disk access could move the bottleneck from CPU to disk speed - especially for people using hard drive instead of SSD.

[wizziwig]

Did some more testing.  When I disable one of my two ISP and drop my total download speed from 55 MByte/s to 43 MByte/s, then the decoding thread can keep up with downloads.  Decoder thread is never more than 2-3 files behind download thread and always stays in the article cache.  Playing with thread priorities doesn't help because I have many idle cores available on this 8-core system.  I guess it would be even worse for people with 1Gbit internet connections unless you have very fast overclocked CPU.

[knoop]

I got the same problem.

Limiting speed down to 150 mbit/s solves the problem, but normally my decoding speed is slower than download rate.

My temp folder is located on a ramdisk, the download folder on a Intel m2 evo ssd.
4 CPU cores @3,6 ghz are definitely NOT busy...

Article Caching is definitely NOT solving the problem. Is there another way for 'bypassing' the decoding progress, to write downloaded data directly into download folder?
 
I would like to have an upgrade to 400 mbit/s internet connection...