Having been using PreUsenet for a short time, I was impressed by how easy it makes the process of preparing files for usenet posting. Had stayed away from posting until recently, but the script makes it painless.
Reading thru the web, I came across a really good page explaining
the optimal settings to use for usenet posting. It made a lot of sense, so I made a spreadsheet to do some calculations, and started working in modding a copy of PreUsenet. It's working well, imo.
Here are some notes on what I did, maybe useful for other users as well?
:: Understanding the logic behind the settings ::-------------------------------------------------------------------------------------------------------
- each post line can have a maximum of 128 characters, so
128 Bytes per post line is our basic number.
- since we are posting large files, we have to use a number of posts to send the data. Larger number of lines allows more data per post, but servers have limits. Depending on what your usenet provider supports, you can use 3000 lines, 5000 lines, even 10000 lines if desired (and supported).
- The
number of lines per post is important - it sets the total Byte size of a full post, called a
segment.
segment size = (number of lines x 128 Bytes per line)- For optimal posting, we should try to make all the RAR and PAR files based on
multiples of the segment size. If a file fits an exact number of segments, we avoid messages with leftover Bytes. Less posts, less headers to handle.
- The size of each RAR and PAR file should then be calculated depending on the user's lines per post. It needs to be rounded to the nearest number of segments (an integer value; no decimals).
:: An example with numbers ::-------------------------------------------------------------------------------------------------------
- user sets 3,000 lines per post. each line is 128 Bytes. A segment = 3,000 x 128 = 384,000 Bytes
- using the SMALL Rars setting in PreUsenet (normal videos, 700 MB or so), the desired size for each RAR is 15 MB.
We can calculate optimal settings like this :
Goal = 15 MB or 15 x 1024 x 1024 , which is 15,728,640 Bytes.
Needed segments = goal size / segment size , which is then rounded.
needed_seg=(15,278,640 / 384,000), which is
40.96. We round this to
41, the closest integer number.
Actual RAR size = required_segments x segment_size = 41 x 384,000 -- resulting in an optimal RAR size of
15,744,000 Bytes.
:: Using Tools to Help ::-------------------------------------------------------------------------------------------------------
Since we have three possible job sizes in PreUsenet (based on SMALL, MKV and BLURAY file sizes), and might want to experiment with line numbers, pretty soon there's a lot of calculating required.
That is well handled by a spreadsheet, like this :
Or by using the nice 'Efficient PAR2 / RAR Calculator'. The screenshot below show the settings for 15 MB RAR, at 3000 lines per post :
Which confims our hand calcs and the spreadsheet as well.
This tool can be
downloaded HERE.
:: Summarizing All That ::-------------------------------------------------------------------------------------------------------
1. Number of yEnc lines is usually 3000 of 5000
2. Segment size = number of yEnc lines * 128 (bytes)
3. Number of segments = desired RAR size / segment size (rounded)
4. RAR size = number of segments * segment size (bytes)
5. Create RAR files with the size found in the previous step
6. PAR2 block size = segment size (bytes)
:: How to Implement this in PreUsenet ::-------------------------------------------------------------------------------------------------------
We can edit the PreUsenet scripts to define the ideal Byte sizes for RAR and PAR2 files, these are hard coded in the '
Settings.bat' file.
You need to know the number of lines your poster tool is using; nowadays,
5000 lines is a good starting point, Astraweb supports up to 11,000 or so.
Even if you want to stay conservative, and use 3,000 lines, setting the RAR and PAR sizes will maximize performance for posting.
Editing Settings.bat - the variables that define RAR and PAR sizes (start on line 17 to the end of the file) :
:: Rar archive size for small Rars (in MB)
SET RARSIZE=15
:: Rar archive size for rarring mkv or 720p (in MB)
SET BIGSIZE=50
:: Rar archive size for rarring Blurays or HDDVD (in MB)
SET BLUSIZE=100
:: Par2 Blocksize for small Posts
SET PAR2BLOCK=768000
:: Par2 Blocksize for mkv or 720p Posts
SET BIGBLOCK=1920000
:: Par2 Blocksize for Bluray or HDDVD Posts
SET BLUBLOCK=1920000
:: My Test script is a lot more verbose, since I left notes with the sizes for different line numbers, etc. This block could replace the standard lines 17 to end of file block of text quoted above :
:: Rar archive size for SMALL Rars, ~15 MB // EDITED for = 5000 lines setting, size in BYTES.
::
:; RAR size DEPENDS on post line count; 3000 lines = 15,744,000 Bytes
:: 5000 lines = 16,000,000 Bytes; 10000 lines = 15,360,000 Bytes
::
SET RARSIZE=16000000
::
:: Rar archive size for MEDIUM MKV or 720p, ~50 MB // EDITED for = 5000 lines setting, size in BYTES.
::
:: RAR size DEPENDS on post line count; 3000 lines = 52,608,000 Bytes
:: 5000 lines = 52,480,000 Bytes; 10000 lines = 52,480,000 Bytes
::
SET BIGSIZE=52480000
::
:: Rar archive size for HUGE Blurays or HDDVD, ~100 MB // EDITED for = 5000 lines setting, size in BYTES.
::
:: RAR size DEPENDS on post line count; 3000 lines = 104,832,000 Bytes
:: 5000 lines = 104,960,000 Bytes; 10000 lines = 104,9600,000 Bytes
::
SET BLUSIZE=104960000
::
:: Par2 Blocksize for small Posts // EDITED for = 5000 lines setting, size in BYTES
::
:: PAR size DEPENDS ONLY on post line count; 3000 lines = 384,000 Bytes
:: 5000 lines = 640,000 Bytes; 10000 lines = 1,280,000 Bytes
:: ** note: original PreUsenet was SET PAR2BLOCK=768000 (2 segments per block)
::
SET PAR2BLOCK=640000
::
:: Par2 Blocksize for mkv or 720p Posts // EDITED for = 5000 lines setting, size in BYTES
:: ** note: original PreUsenet was SET BIGBLOCK=1920000 (3 segments per block)
::
SET BIGBLOCK=640000
::
:: Par2 Blocksize for Bluray or HDDVD Posts // EDITED for = 5000 lines setting, size in BYTES
:: ** note: original PreUsenet was SET BLUBLOCK=1920000 (3 segments per block)
::
SET BLUBLOCK=640000
:: ** We are
NOT done yet - we changed the RAR sizes from
MegaBytes to
Bytes, and need to modify the RAR creation command, to allow for that.
cr4zyfr4g mentioned the exact line numbers needed for this in a forum post, when someone asked.
Q. Is it possible to configure the rarsize in bytes instead of MB?
A. yes you can change line 259 and 269 and replace the M from "-v%RARSIZE%M" to B
So, open PreUsenet.bat on an editor and do those fixes.
Existing code :
line 259 :
"%INSTALLPATH%Rar.exe" "a" "-v%RARSIZE%M" "-m0" "-y" "-ep1" "-ed" "-vn" "%TARGETLOCATION%\%FILENAME%.rar" "%PATH%%FILENAME%%EXT%" |
and then line 269 :
"%INSTALLPATH%Rar.exe" "a" "-v%RARSIZE%M" "-m0" "-y" "-ep1" "-x*.nfo" "-r" "-vn"
Change to :
line 259 :
"%INSTALLPATH%Rar.exe" "a" "-v%RARSIZE%B" "-m0" "-y" "-ep1" "-ed" "-vn" "%TARGETLOCATION%\%FILENAME%.rar" "%PATH%%FILENAME%%EXT%" |
and then line 269 :
"%INSTALLPATH%Rar.exe" "a" "-v%RARSIZE%B" "-m0" "-y" "-ep1" "-x*.nfo" "-r" "-vn"
Now, that's it.
:: Suggestions ::-------------------------------------------------------------------------------------------------------
- keep your existing PreUsenet folder as it is. Make a copy of the WHOLE folder, paste it somewhere else, rename to clearly mark a Test version.
- Rename the PreUsenet.bat, mine is called MOD_PreUsenet.bat.
- Edit the copy on the test folder only.
- then make a shortcut to it on the SendTo location. Now you have two different versions, the standard release and a modded one. Test and see how it works, you did not break the original this way.
cr4zyfr4g uses larger PAR2 block sizes in PreUsenet, the default is to use PAR block size == Segment size. He mentioned that using one single segment per PAR block slows down par file creation, and that seems to be true.
There are notes in the modded version of Settings.bat above, mentioning the default PAR block sizes; you can experiment and find what works best for you.
Advantage of smaller PAR block - any damage to a post, even a single Byte being mangled, will require rebuilding with PAR. If the par block is larger than a segment (a single post) the rest of the recovery data in that PAR block gets wasted, from what I have read.
Hope this helps, any questions or comments are welcome.
