Author Topic: Using customized RAR and PAR block size for optimal posting  (Read 27717 times)

blitze

  • Guest
Using customized RAR and PAR block size for optimal posting
« on: October 09, 2009, 05:27:04 am »
Having been using PreUsenet for a short time, I was impressed by how easy it makes the process of preparing files for usenet posting. Had stayed away from posting until recently, but the script makes it painless. :)

Reading thru the web, I came across a really good page explaining the optimal settings to use for usenet posting. It made a lot of sense, so I made a spreadsheet to do some calculations, and started working in modding a copy of PreUsenet. It's working well, imo.

Here are some notes on what I did, maybe useful for other users as well?


::  Understanding the logic behind the settings  ::
-------------------------------------------------------------------------------------------------------

- each post line can have a maximum of 128 characters, so 128 Bytes per post line is our basic number.

- since we are posting large files, we have to use a number of posts to send the data. Larger number of lines allows more data per post, but servers have limits. Depending on what your usenet provider supports, you can use 3000 lines, 5000 lines, even 10000 lines if desired (and supported).

- The number of lines per post is important - it sets the total Byte size of a full post, called a segment.

segment size = (number of lines x 128 Bytes per line)

- For optimal posting, we should try to make all the RAR and PAR files based on multiples of the segment size. If a file fits an exact number of segments, we avoid messages with leftover Bytes. Less posts, less headers to handle.

- The size of each RAR and PAR file should then be calculated depending on the user's lines per post. It needs to be rounded to the nearest number of segments (an integer value; no decimals).


::  An example with numbers  ::
-------------------------------------------------------------------------------------------------------

- user sets 3,000 lines per post. each line is 128 Bytes. A segment = 3,000 x 128 = 384,000 Bytes

- using the SMALL Rars setting in PreUsenet (normal videos, 700 MB or so), the desired size for each RAR is 15 MB.

We can calculate optimal settings like this :

Goal = 15 MB or 15 x 1024 x 1024 , which is  15,728,640 Bytes.

Needed segments = goal size / segment size , which is then rounded.

needed_seg=(15,278,640 / 384,000), which is 40.96. We round this to 41, the closest integer number.

Actual RAR size = required_segments x segment_size = 41 x 384,000 -- resulting in an optimal RAR size of 15,744,000 Bytes.


::  Using Tools to Help  ::
-------------------------------------------------------------------------------------------------------

Since we have three possible job sizes in PreUsenet (based on SMALL, MKV and BLURAY file sizes), and might want to experiment with line numbers, pretty soon there's a lot of calculating required.

That is well handled by a spreadsheet, like this :



Or by using the nice 'Efficient PAR2 / RAR Calculator'.  The screenshot below show the settings for 15 MB RAR, at 3000 lines per post :



Which confims our hand calcs and the spreadsheet as well. ;) This tool can be downloaded HERE.


::  Summarizing All That  ::
-------------------------------------------------------------------------------------------------------

1. Number of yEnc lines is usually 3000 of 5000
2. Segment size = number of yEnc lines * 128 (bytes)
3. Number of segments = desired RAR size / segment size (rounded)
4. RAR size = number of segments * segment size (bytes)
5. Create RAR files with the size found in the previous step
6. PAR2 block size = segment size (bytes)


::  How to Implement this in PreUsenet  ::
-------------------------------------------------------------------------------------------------------

We can edit the PreUsenet scripts to define the ideal Byte sizes for RAR and PAR2 files, these are hard coded in the 'Settings.bat' file.

You need to know the number of lines your poster tool is using; nowadays, 5000 lines is a good starting point, Astraweb supports up to 11,000 or so.

Even if you want to stay conservative, and use 3,000 lines, setting the RAR and PAR sizes will maximize performance for posting.


Editing Settings.bat - the variables that define RAR and PAR sizes (start on line 17 to the end of the file) :

Code: [Select]
:: Rar archive size for small Rars (in MB)
SET RARSIZE=15
:: Rar archive size for rarring mkv or 720p (in MB)
SET BIGSIZE=50
:: Rar archive size for rarring Blurays or HDDVD (in MB)
SET BLUSIZE=100
:: Par2 Blocksize for small Posts
SET PAR2BLOCK=768000
:: Par2 Blocksize for mkv or 720p Posts
SET BIGBLOCK=1920000
:: Par2 Blocksize for Bluray or HDDVD Posts
SET BLUBLOCK=1920000
::

My Test script is a lot more verbose, since I left notes with the sizes for different line numbers, etc. This block could replace the standard lines 17 to end of file block of text quoted above :

Code: [Select]
:: Rar archive size for SMALL Rars, ~15 MB // EDITED for = 5000 lines setting, size in BYTES.
::
:; RAR size DEPENDS on post line count; 3000 lines = 15,744,000 Bytes
:: 5000 lines = 16,000,000 Bytes; 10000 lines = 15,360,000 Bytes
::
SET RARSIZE=16000000
::
:: Rar archive size for MEDIUM MKV or 720p, ~50 MB // EDITED for = 5000 lines setting, size in BYTES.
::
:: RAR size DEPENDS on post line count; 3000 lines = 52,608,000 Bytes
:: 5000 lines = 52,480,000 Bytes; 10000 lines = 52,480,000 Bytes
::
SET BIGSIZE=52480000
::
:: Rar archive size for HUGE Blurays or HDDVD, ~100 MB // EDITED for = 5000 lines setting, size in BYTES.
::
:: RAR size DEPENDS on post line count; 3000 lines = 104,832,000 Bytes
:: 5000 lines = 104,960,000 Bytes; 10000 lines = 104,9600,000 Bytes
::
SET BLUSIZE=104960000
::
:: Par2 Blocksize for small Posts // EDITED for = 5000 lines setting, size in BYTES
::
:: PAR size DEPENDS ONLY on post line count; 3000 lines = 384,000 Bytes
:: 5000 lines = 640,000 Bytes; 10000 lines = 1,280,000 Bytes
:: ** note: original PreUsenet was SET PAR2BLOCK=768000 (2 segments per block)
::
SET PAR2BLOCK=640000
::
:: Par2 Blocksize for mkv or 720p Posts // EDITED for = 5000 lines setting, size in BYTES
:: ** note: original PreUsenet was SET BIGBLOCK=1920000 (3 segments per block)
::
SET BIGBLOCK=640000
::
:: Par2 Blocksize for Bluray or HDDVD Posts // EDITED for = 5000 lines setting, size in BYTES
:: ** note: original PreUsenet was SET BLUBLOCK=1920000 (3 segments per block)
::
SET BLUBLOCK=640000
::


** We are NOT done yet - we changed the RAR sizes from MegaBytes to Bytes, and need to modify the RAR creation command, to allow for that.

cr4zyfr4g mentioned the exact line numbers needed for this in a forum post, when someone asked. :)

Quote
Q. Is it possible to configure the rarsize in bytes instead of MB?

A. yes you can change line 259 and 269 and replace the M from "-v%RARSIZE%M" to B

So, open PreUsenet.bat on an editor and do those fixes.

Existing code :

Quote
line 259 :
"%INSTALLPATH%Rar.exe" "a" "-v%RARSIZE%M" "-m0" "-y" "-ep1" "-ed" "-vn" "%TARGETLOCATION%\%FILENAME%.rar" "%PATH%%FILENAME%%EXT%" |

and then line 269 :
"%INSTALLPATH%Rar.exe" "a" "-v%RARSIZE%M" "-m0" "-y" "-ep1" "-x*.nfo" "-r" "-vn"

Change to :

Quote
line 259 :
"%INSTALLPATH%Rar.exe" "a" "-v%RARSIZE%B" "-m0" "-y" "-ep1" "-ed" "-vn" "%TARGETLOCATION%\%FILENAME%.rar" "%PATH%%FILENAME%%EXT%" |

and then line 269 :
"%INSTALLPATH%Rar.exe" "a" "-v%RARSIZE%B" "-m0" "-y" "-ep1" "-x*.nfo" "-r" "-vn"

Now, that's it. :)


::  Suggestions ::
-------------------------------------------------------------------------------------------------------

- keep your existing PreUsenet folder as it is. Make a copy of the WHOLE folder, paste it somewhere else, rename to clearly mark a Test version.

- Rename the PreUsenet.bat, mine is called MOD_PreUsenet.bat.

- Edit the copy on the test folder only.

- then make a shortcut to it on the SendTo location. Now you have two different versions, the standard release and a modded one. Test and see how it works, you did not break the original this way. ;)

cr4zyfr4g uses larger PAR2 block sizes in PreUsenet, the default is to use PAR block size == Segment size. He mentioned that using one single segment per PAR block slows down par file creation, and that seems to be true. :(

There are notes in the modded version of Settings.bat above, mentioning the default PAR block sizes; you can experiment and find what works best for you.

Advantage of smaller PAR block - any damage to a post, even a single Byte being mangled, will require rebuilding with PAR. If the par block is larger than a segment (a single post) the rest of the recovery data in that PAR block gets wasted, from what I have read.

Hope this helps, any questions or comments are welcome. ;)


Offline cr4zyfr4g

  • Global Moderator
  • *****
  • Posts: 781
  • German n00b
Re: Using customized RAR and PAR block size for optimal posting
« Reply #1 on: October 09, 2009, 12:50:58 pm »
very nice post

thx for all your hard work there :)

blitze

  • Guest
Re: Using customized RAR and PAR block size for optimal posting
« Reply #2 on: October 10, 2009, 03:10:30 am »
Thanks, cr4zyfr4g. :)

The excellent page at Binaries4All on post size optimizing was the source of most of the information. Their other pages for posting, creating NFO, PAR files and more are great reading too. I learned a lot from them.

blitze

  • Guest
Re: Using customized RAR and PAR block size for optimal posting
« Reply #3 on: October 15, 2009, 12:52:50 am »
Looking inside an NZB file in a text editor today, I can see clearly the difference between a less optimal setup (generic NZB, someone's post at alt.bin.tv) and a file packaged up using the calculations posted above.

First the random nzb from the wild :

Code: [Select]
<segments>
<segment bytes="660829" number="1">4ad1ff99$1$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="660655" number="2">4ad1ff9c$0$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="661093" number="3">4ad1ff9e$0$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="661110" number="4">4ad1ffa1$0$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="660772" number="5">4ad1ffa3$0$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="660629" number="6">4ad1ffa5$0$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="660272" number="7">4ad1ffa8$1$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="660186" number="8">4ad1ffaa$1$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="660577" number="9">4ad1ffad$1$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="660608" number="10">4ad1ffaf$0$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="660714" number="11">4ad1ffb2$0$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="660674" number="12">4ad1ffb4$1$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="660666" number="13">4ad1ffb7$0$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="661026" number="14">4ad1ffb9$3$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="661161" number="15">4ad1ffbb$1$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="661418" number="16">4ad1ffbe$0$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="660754" number="17">4ad1ffc0$1$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="660725" number="18">4ad1ffc3$0$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="660747" number="19">4ad1ffc5$1$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="660597" number="20">4ad1ffc8$0$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="660188" number="21">4ad1ffca$1$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="660831" number="22">4ad1ffcd$1$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="661117" number="23">4ad1ffcf$1$10482$2d805a3e@uploadreader.eweka.nl</segment>
<segment bytes="265120" number="24">4ad1ffd2$0$10482$2d805a3e@uploadreader.eweka.nl</segment>
</segments>

Notice the variation of the segment sizes (individual posts, with varying numbers of lines each).

And the one with RAR sizes optimal :

Code: [Select]
<segments>
<segment number="1" bytes="640000">4ad20b48$1$10482$2d805a3e@thefish.net.local</segment>
<segment number="2" bytes="640000">4ad20b4a$1$10482$2d805a3e@thefish.net.local</segment>
<segment number="3" bytes="640000">4ad20b4d$0$10482$2d805a3e@thefish.net.local</segment>
<segment number="4" bytes="640000">4ad20b4f$2$10482$2d805a3e@thefish.net.local</segment>
<segment number="5" bytes="640000">4ad20b51$1$10482$2d805a3e@thefish.net.local</segment>
<segment number="6" bytes="640000">4ad20b54$1$10482$2d805a3e@thefish.net.local</segment>
<segment number="7" bytes="640000">4ad20b56$0$10482$2d805a3e@thefish.net.local</segment>
<segment number="8" bytes="640000">4ad20b58$3$10482$2d805a3e@thefish.net.local</segment>
<segment number="9" bytes="640000">4ad20b59$3$10482$2d805a3e@thefish.net.local</segment>
<segment number="10" bytes="640000">4ad20b5a$3$10482$2d805a3e@thefish.net.local</segment>
<segment number="11" bytes="640000">4ad20b5b$3$10482$2d805a3e@thefish.net.local</segment>
<segment number="12" bytes="640000">4ad20b5c$3$10482$2d805a3e@thefish.net.local</segment>
<segment number="13" bytes="640000">4ad20b5d$3$10482$2d805a3e@thefish.net.local</segment>
<segment number="14" bytes="640000">4ad20b5e$3$10482$2d805a3e@thefish.net.local</segment>
<segment number="15" bytes="640000">4ad20b5f$3$10482$2d805a3e@thefish.net.local</segment>
<segment number="16" bytes="640000">4ad20b60$3$10482$2d805a3e@thefish.net.local</segment>
<segment number="17" bytes="640000">4ad20b61$3$10482$2d805a3e@thefish.net.local</segment>
<segment number="18" bytes="640000">4ad20b62$3$10482$2d805a3e@thefish.net.local</segment>
<segment number="19" bytes="640000">4ad20b63$3$10482$2d805a3e@thefish.net.local</segment>
<segment number="20" bytes="640000">4ad20b64$3$10482$2d805a3e@thefish.net.local</segment>
<segment number="21" bytes="640000">4ad20b65$3$10482$2d805a3e@thefish.net.local</segment>
<segment number="22" bytes="640000">4ad20b66$3$10482$2d805a3e@thefish.net.local</segment>
<segment number="23" bytes="640000">4ad20b67$3$10482$2d805a3e@thefish.net.local</segment>
<segment number="24" bytes="640000">4ad20b68$3$10482$2d805a3e@thefish.net.local</segment>
<segment number="25" bytes="640000">4ad20b69$3$10482$2d805a3e@thefish.net.local</segment>
</segments>

I like the elegance of the second example - 25 segments, all the same exact size (640,000 Bytes each), making the full block size 25 x 640,000 = 16,000,000 Bytes.

That's my chosen setting in PreUsenet, optimal for 5000 lines posting.

« Last Edit: October 15, 2009, 02:05:31 am by blitze »

Offline Hecks

  • Contributor
  • ***
  • Posts: 2011
  • naughty cop
Re: Using customized RAR and PAR block size for optimal posting
« Reply #4 on: October 15, 2009, 08:38:35 am »
Well, do bear in mind that there'll be a difference in reported segment size depending on whether the NZB is made by the poster or made by an indexer.  For the latter, the info is taken from the article headers, and this varies between servers.  Also, the yEnc headers & footers in the message are a factor, not sure if they're included in your calculations.

blitze

  • Guest
Re: Using customized RAR and PAR block size for optimal posting
« Reply #5 on: October 15, 2009, 11:38:25 am »
Hecks, thanks for pointing that out.

It explains the odd segment sizes on the sample NZB shown in the post above, couldn't understand how the segments in it varied from one post to the next, and size did not divide cleanly by 128 Bytes as expected. Live and learn.  :(

Looked at a quick sample of one I generated locally :



Versus the same part, from a NFO search result :



And there are the same odd variations here too. :(

The calculations in my original post only create the size for RAR piece as a integer multiple of segment sizes; segment sizes being user-selected, people deciding how many lines per post to use.

yEnc post headers and footers add the extra Bytes in the example from the wild, as you pointed. (more for me to think about)

Offline Rdl

  • Administrator
  • *****
  • Posts: 4074
Re: Using customized RAR and PAR block size for optimal posting
« Reply #6 on: October 15, 2009, 12:16:00 pm »
Nothing odd about it. As Hecks pointed out you're confusing bytes before yenc encoding and after encoding+yenc header/footer

brainz101

  • Guest
Re: Using customized RAR and PAR block size for optimal posting
« Reply #7 on: January 14, 2010, 06:35:59 am »
Thank you, I found this to very helpful information! :)