pyReScene false positive on -mt[n] option

Summary:

-mt1 instead of -mt4 for instance

pyReScene false positive on -mt[n] option

mechtex (guest) 20 Sep 2013 00:11

Hello,

I found that you might run into some edge case where incorrect positive commandline flags are retained.
Release in question is Accel_World_Kasoku_no_Chouten_JPN_PSP-PLAYASiA.
With no modding, pyRescene attempts to rebuild the file with -mt1 and raises a ValueError("Still not fine :(.")

Here are the results from a reconstruct run attempt. I slightly modded main.py to dump the invoked commandline and to purposely fail detection.

O:\>python.exe "C:\Tools\pyReScene-0.5.1\bin\srr" -z C:\Tools\pyReScene-0.5.1\rar -t E:\tmp Accel_World_Kasoku_no_Chouten_JPN_PSP-PLAYASiA.srr
 
SRR file created with pyReScene Auto 0.4.
Recreating stored file: pa-awkncj.nfo
Recreating stored file: pa-awkncj.sfv
Re-creating RAR file: pa-awkncj.rar
Trying to rebuild compressed file pa-awkncj.iso.
Grabbing large enough data piece size for testing.
Trying 2012-06-09 4.20.
[DEBUG] returning False instead of True
* Candidate for matching cmdline: C:\Tools\pyReScene-0.5.1\rar\2012-06-09_rar420.exe a -m5 -mdG -s- -ds -mt1 -vn -o+ -ep -idcd -vn -v19654967b E:\tmp\tmprjjkoa_pyReScene\pyReScene_compressed.rar E:\tmp\tmprjjkoa_pyReScene\pyReScene_data_piece.iso
[DEBUG] returning False instead of True
* Candidate for matching cmdline: C:\Tools\pyReScene-0.5.1\rar\2012-06-09_rar420.exe a -m5 -mdG -s- -ds -mt2 -vn -o+ -ep -idcd -vn -v19654967b E:\tmp\tmprjjkoa_pyReScene\pyReScene_compressed.rar E:\tmp\tmprjjkoa_pyReScene\pyReScene_data_piece.iso
[DEBUG] returning False instead of True
* Candidate for matching cmdline: C:\Tools\pyReScene-0.5.1\rar\2012-06-09_rar420.exe a -m5 -mdG -s- -ds -mt4 -vn -o+ -ep -idcd -vn -v19654967b E:\tmp\tmprjjkoa_pyReScene\pyReScene_compressed.rar E:\tmp\tmprjjkoa_pyReScene\pyReScene_data_piece.iso
[DEBUG] returning False instead of True
* Candidate for matching cmdline: C:\Tools\pyReScene-0.5.1\rar\2012-06-09_rar420.exe a -m5 -mdG -s- -ds -mt8 -vn -o+ -ep -idcd -vn -v19654967b E:\tmp\tmprjjkoa_pyReScene\pyReScene_compressed.rar E:\tmp\tmprjjkoa_pyReScene\pyReScene_data_piece.iso
Grabbing large enough data piece size for testing.
Trying 2012-06-09 4.20.
 
(--snipped: 2nd attempt for next_block case--)
 
No good RAR version found.
No good RAR version found.

You can see that several switches do match.
When you manually run the -mt1 commandline AND specifying -v20000000b, you find that several archives seem to be fine (good CRC w.r.t sfv): .rar up to .r02 (ca. 80 MB of compressed data)

Does that confuse pyRescene into thinking -mt1 should be perfectly fine?
The actual correct switch is -mt4, and if you pass detection only for that flag, reconstruction works fine.

Here is what the rar.txt manual states:

    -mt<threads>
            Set the number of threads. Available in Windows version only.

            <threads> parameter can take values from 0 to 16.
            It defines the recommended number of active threads
            for compression algorithm. If it is greater than 0,
            RAR will use the multithreaded version of compression
            algorithm providing higher speed on multiprocessor
            architectures. Real number of active threads can differ
            from the specified.

            If <threads> is zero, RAR will use the single threaded
            compression algorithm.

            Change of <threads> parameter slightly affects the compression
            ratio, so archives created with different -mt switches
            will not be exactly the same even if all other compression
            settings are equal.

            If -mt switch is not specified, RAR will try to detect
            the number of available processors and select the optimal
            number of threads automatically.

Re: pyReScene false positive on -mt[n] option

Gfy 21 Sep 2013 19:42

Does that confuse pyRescene into thinking -mt1 should be perfectly fine?

Yes.

The algorithm works/fails as follows:

• When having a large file, it will grab a piece of that file, large enough, that when compressed, it will still be enough to create all the data of the first RAR volume.
• The different RAR versions will be run on this piece of data to determine if they match. This is done based on the CRC that is stored inside the RAR header in the SRR and a checksum of the generated data. An educated guess is done based on the file date of the archived file to determine which RAR version to try first. (All this is done to speed up the process)
• When the checksums match, it assumes to have found the good RAR version. After all, the first RAR will be able to reconstruct with the generated data. But like you found out, the version is right, but the settings aren't.
• It compresses the data.
• Before it returns a file like object to be used by the reconstruction algorithm, it does a last quick test to see if the reconstruction will succeed or not. This is the moment when "Still not fine :(." will be thrown. (it compares the size of expected compressed data with the size of the compressed data we just created)

A solution will be to try an other setting when this is detected.

But then this could still be an issue:
http://rescene.wikidot.com/forum/t-684654/europa-universalis-iv-flt

"Still not fine :(." isn't thrown (the sizes match), but the output isn't correct either. This case won't be so easy to detect.