For CODEX releases I was able to use 2016-08-15_rar540.exe no problem. But now when they're gone some major releases show up from other groups. For FairLight I haven't found a RAR version that works. Anybody know of a version that is compatible with their releases? Are they using something as ancient as themselves? ;)
Might as well ask the same regarding Razor 1911.
The problem with FLT is that they seem to use the macos version of RAR. See e.g. the HOST_OS in in this recent release:
Block: RAR File; offset: 0x23B9 (9145 bytes)
|Header bytes: 9d4374c28159007fc1eb0bffffffff04c98efcfda6226c531d3131000000000000000000ffffffff666c742d6173736574746f5f636f7273615f636f6d706574697a696f6e655f616d65726963616e5f747261636b2e69736f
|HEAD_CRC: 0x439D
|HEAD_TYPE: 0x74 (RAR File)
|HEAD_FLAGS: 0x81C2
| 0x8000 LONG_BLOCK (ADD_SIZE field present)
| 0x0002 LHD_SPLIT_AFTER (file continued in next volume)
| 0x0100 LHD_LARGE (only used for files larger then 2GiB)
| 0x00C0 LHD_WINDOW (Dictionary size 4096 KiB)
|HEAD_SIZE: 0x59 (89 bytes)
+PACK_SIZE: 199999871 bytes (ADD_SIZE + HIGH_PACK_SIZE field)
+UNP_SIZE: 18446744073709551615 bytes
+HOST_OS: Mac OS used to create this file block.
+FILE_CRC: FDFC8EC9
+FTIME: 2021-11-12 04:21:12
+UNP_VER: Version 2.9 is needed to extract.
+METHOD: Fastest compression
+NAME_SIZE: always present
+ATTR: 0
+HIGH_PACK_SIZE: 0
+HIGH_UNP_SIZE: 4294967295
+FILE_NAME: flt-assetto_corsa_competizione_american_track.isoI don't know how to reconstruct this on a non-mac OS :-(
Razor I haven't yet found problematic, do you have an example of a release that fails?
Thank you for the reply! That I've not noticed until now… :)
That's very interesting about MacOS. And unfortunate. Will be hard to recreate for me then.
As for Razor 1911, having the same type of problems rebuilding the release as with FairLight. But maybe I just haven't selected enough RAR versions. Here is a more recent release I'm having problems with:
https://www.srrdb.com/release/details/A_Plague_Tale_Requiem_v1.3.0.0-Razor1911
It indeed does not seem to recreate automatically using pyrescene. However I was able to rescene it manually.
I'm not entirely sure, but I feel like the problem has something to do with the volume size. I.e. a similar case to the packing method depending on the volume size. Although in this case it's only a single file, so it's not quite that.
There is some logic in rar that decides on when to use the "Extended time field", and when that field is present the file header is a bit larger, leaving less room for compressed data with the fixed volume sizes. So some of the compressed bytes that should go at the end of volume A end up at the start of volume B, and sometimes this make a difference for the final compressed output it seems.
To set the correct timestamp on the file I used:
touch --no-create --time=modify --date 2005-05-13T18:57:29 rzr-aplaguetalerequiem1300.isoSo after getting the amount of compressed data relative to the amount of headers correct it now reconstructs with the following command line:
2015-11-18_rar530.exe a -m1 -mdG -v250000000b -s- -ds -mt12 -vn -o+ -ep -idcd -ma4 rzr-aplaguetalerequiem1300.rar aplaguetalerequiem1300.isoHowever pyrescene uses a seemingly random value for the volume size, 189417906b in this case, so even if the correct timestamp is set on the file, it still won't be a match even if it uses the right rar version and -mt argument. And then there seems to be yet another difference that I can't explain, because after forcing pyrescene to use the correct volume size it still doesn't detect the correct rar version from the sample file, so perhaps the actual size of the input file also plays a role? Probably because LHD_LARGE doesn't get set, which again also affects the size of the headers by omitting some fields (leaving more room for compressed data).
Started watching that issue now to see if something happens in the future. :)
I tried to do what you did. Don't have the touch command on Windows, but ran this in PowerShell command which supposedly does the same thing:
(Get-Item "rzr-aplaguetalerequiem1300.iso").LastWriteTime=("13 May 2005 18:57:29")What I'm wondering though is how did you arrive at this date? And for that matter, the information you posted back in August, how do you get that for a release?
Regardless, I haven't had to mess around with modified dates for e.g. CODEX releases when recreating them with pyReScene in the past. Is that really affecting the resulting checksums? Or maybe it was something else in the header that was a bit too large and using touch you just reset it sort of? The date doesn't really matter? And the PowerShell command does not do "enough" here?
Also, how were you able to determine that 5.30 was going to be the correct version in this case?
Anyway, after having changed the modify date, I ran your command above and it happily created the volumes, but for me all the checksums were wrong still. Maybe I misunderstood you, weren't you able to reacreate the proper files manually above? I still failed at least. :)
What I'm wondering though is how did you arrive at this date? And for that matter, the information you posted back in August, how do you get that for a release?
You can see the technical information stored in a .srr file using the —details (-e) switch. This will output the stored RAR headers in a human readable format. It is a lot of output, so you probably want to redirect the output into a text file:
srr --details A_Plague_Tale_Requiem_v1.3.0.0-Razor1911.srr > A_Plague_Tale_Requiem_v1.3.0.0-Razor1911.txtAfter opening this in a text editor you can look at e.g. the first RAR File block. For this release it's:
Block: RAR File; offset: 0x2B87 (11143 bytes)
|Header bytes: 30fe74c291480010b2e60e000084ed020b116d032e97ad321d311e0020000000000000000b000000727a722d61706c6167756574616c657265717569656d313330302e69736f00c0
|HEAD_CRC: 0xFE30
|HEAD_TYPE: 0x74 (RAR File)
|HEAD_FLAGS: 0x91C2
| 0x8000 LONG_BLOCK (ADD_SIZE field present)
| 0x0002 LHD_SPLIT_AFTER (file continued in next volume)
| 0x0100 LHD_LARGE (only used for files larger then 2GiB)
| 0x1000 LHD_EXTTIME (Extended time field present)
| 0x00C0 LHD_WINDOW (Dictionary size 4096 KiB)
|HEAD_SIZE: 0x48 (72 bytes)
+PACK_SIZE: 249999888 bytes (ADD_SIZE + HIGH_PACK_SIZE field)
+UNP_SIZE: 51229491200 bytes
+HOST_OS: Windows used to create this file block.
+FILE_CRC: 036D110B
+FTIME: 2005-05-13 18:57:28
+UNP_VER: Version 2.9 is needed to extract.
+METHOD: Fastest compression
+NAME_SIZE: always present
+ATTR: 20
+HIGH_PACK_SIZE: 0
+HIGH_UNP_SIZE: 11
+FILE_NAME: rzr-aplaguetalerequiem1300.iso
+modification time: 2005-05-13 18:57:29
+creation time: UNKNOWN
+last access time: UNKNOWN
+archival time: UNKNOWNFrom this we see that:
Also, how were you able to determine that 5.30 was going to be the correct version in this case?
In this case, a lucky guess. Most recent razor releases use this version with either -mt12 or -mt16. Determining the correct rar version is always trial and error :-(.
Regardless, I haven't had to mess around with modified dates for e.g. CODEX releases when recreating them with pyReScene in the past. Is that really affecting the resulting checksums? Or maybe it was something else in the header that was a bit too large and using touch you just reset it sort of? The date doesn't really matter? And the PowerShell command does not do "enough" here?
Anyway, after having changed the modify date, I ran your command above and it happily created the volumes, but for me all the checksums were wrong still. Maybe I misunderstood you, weren't you able to reacreate the proper files manually above? I still failed at least. :)
It's strange that it fails for you, perhaps something else is still not quite right. For me it recreated all .rar files exactly (correct checksums) with the above command, after setting the correct date.
I don't know why it's the date that seems to trigger this, I have observed this with other release too. You can also try to manually control which timestamps get included and with what precision using the -ts<m,c,a>[N] switch(es).
You will have to look at the difference in the generated rar header vs. what is expected (from the rar header stored in the srr). Unfortunately I don't have an easy way (i.e. tool) for you to this, other than using srr to generate a .srr file for your custom rar, and outputting the headers with the —details switch. Then compare the fields of the two headers, and see which one differs. If you do this and you use a 500MB sample file (so you don't have to rar the whole release for every test), then be aware that using a smaller sample file will affect the LHD_LARGE flag and its related fields (HIGH_PACK_SIZE and HIGH_UNP_SIZE), and therefore potentially the amount of compressed data and checksums.
One easy thing you can still try though is the following. Now that I am looking at the .iso I have here, it has both the created and the modified timestamps set to 2005-05-13 18:57:29. I did do this by-hand, so perhaps at some point I also set the created timestamp (this should be FTIME), and I just forgot. So try to set both to the same value. If it it turns out that this is it, it's a bit of a coincidence this worked because the actual timestamps differ by 1 second (FTIME is 28 seconds, while the modified time is 29 seconds). However, FTIME actually uses the DOS timestamp format, which is only accurate to 2 seconds, so the 29 gets rounded down to 28 when stored in the rar header, making it so that it still matches the .srr…
Wow, that's a lot to digest! There's a lot to this.
However, FTIME actually uses the DOS timestamp format, which is only accurate to 2 seconds, so the 29 gets rounded down to 28 when stored in the rar header, making it so that it still matches the .srr…
… my head is hurting. :'D
Anyway, I set created date to the same as the modified date using this command in PowerShell:
(Get-Item "rzr-aplaguetalerequiem1300.iso").CreationTime=("13 May 2005 18:57:29")Both dates show the same thing now. But it still didn't work.
I noticed that the checksum of the first volume (rzr-aplaguetalerequiem1300.rar) turned out the same both times, so the date(s?) don't seem to be affecting the end result. I'm getting 7aae9836, but it's supposed to be 3fcff4ef.
However, I did generate an .srr file from the latest attempt now and compared to the one from srrDB. There are a lot of differences, like mine doesn't have the stored .nfo or .sfv files and the application name in the beginning is "pyReScene 0.7", not "pyReScene Auto 0.7" like the real one has.
But I don't think this matters really here, right? We're only interested in the "RAR File" block where the .iso is? And there is a difference. I have:
+ATTR: 220compared to this in the original:
+ATTR: 20What does that mean?
But I don't think this matters really here, right? We're only interested in the "RAR File" block where the .iso is? And there is a difference. I have:
Exactly right :-). It looks like you're really close, the only remaining difference is indeed that ATTR, everything else is the same now and you can even see that the date-time is picked up correctly, you have both the FTIME and the modification time entries matching (so the date and time do end up in the archive, and so they do matter). The checksums will match once all the header data is the same.
What does that mean?
Good question :p. I had to trace this a bit, but with a bit of guessing and reasoning I think these are the raw "file attribute" values that windows uses for the file. I.e. looking at the extraction code, near the end of CmdExtract::ExtractCurrentFile(), you can see how Arc.FileHead.FileAttr is used in a call to SetFileAttr(), which on windows is implemented using SetFileAttributes(). So the ATTR should correspond to the windows file attributes.
The desired value is 0x20, which corresponds to just FILE_ATTRIBUTE_ARCHIVE.
You have 0x220, which corresponds to FILE_ATTRIBUTE_ARCHIVE|FILE_ATTRIBUTE_SPARSE_FILE, I think.
So it seems like the file you have has some strange attribute set (FILE_ATTRIBUTE_SPARSE_FILE), which I don't quite know how to control. "Normal" attributes you can set/unset from a file's properties dialogue (Attributes: [ ] Read-only [ ] Hidden and the Advanced… button next to that). FILE_ATTRIBUTE_SPARSE_FILE seems to have something to do with the way the file is stored? This page states the following:
Sparse file. A file that is a sparse file. A sparse file has an attribute that causes the I/O subsystem to allocate only meaningful (nonzero) data. Nonzero data is allocated on disk, and non-meaningful data (large strings of data composed of zeros) is not.
The documentation for the function that controls this bit mentions that at least in the past (Windows Server 2003 and Windows XP) The only way to clear this attribute is to overwrite the file (for example, by calling the CreateFile function with the CREATE_ALWAYS flag). This suggest that you may try copying the file as the easiest way to clear this bit, and a quick test seems to confirm this:
$ fsutil sparse queryflag test.bin
This file is NOT set as sparse
$ fsutil sparse setflag test.bin
$ fsutil sparse queryflag test.bin
This file is set as sparse
$ copy test.bin test_copy.bin
1 file(s) copied.
$ fsutil sparse queryflag test_copy.bin
This file is NOT set as sparse
$ fsutil usn readdata test.bin
Major Version : 0x3
Minor Version : 0x0
FileRef# : 0x0000000000000000000800000010c4b3
Parent FileRef# : 0x0000000000000000002500000009dfbc
Usn : 0x00000007ba386e10
Time Stamp : 0x0000000000000000 00:00:00 01/01/01
Reason : 0x0
Source Info : 0x0
Security Id : 0x0
File Attributes : 0x220
File Name Length : 0x10
File Name Offset : 0x4c
FileName : test.bin
$ fsutil usn readdata test_copy.bin
Major Version : 0x3
Minor Version : 0x0
FileRef# : 0x0000000000000000001d00000009734f
Parent FileRef# : 0x0000000000000000002500000009dfbc
Usn : 0x00000007ba3877b8
Time Stamp : 0x0000000000000000 00:00:00 01/01/01
Reason : 0x0
Source Info : 0x0
Security Id : 0x0
File Attributes : 0x20
File Name Length : 0x1a
File Name Offset : 0x4c
FileName : test_copy.binThat did it! Copying the 50GB file cleared that attribute as theorized. That of course meant a new create date, so I ran the PowerShell command for that again and double checked that the sparse attribute had not returned and it hadn't, so both dates were in the past and the sparse file attribute was non-present.
And then your command above generated volumes with the correct checksums. :)
The reason that sparse file is set on the file I think comes from how I got the file, which is from qBittorrent, which in turn is based on libtorrent. Apparently sparse file is by default on to (I guess) save on actual writes to the drive, since large strings of zeroes are actually not written? However "Size" and "Size on disk" are both still the full number of bytes (51 229 491 200) for both the sparse, downloaded file and the non-sparse, copied file, so maybe there was actually nothing to be "sparse with" on this particular file, so it had to write the whole thing. Only the attribute is set, without any actual effect on the file itself.
And maybe there seldom is? An .iso like this probably uses some form of internal compression, so long strings of zeroes are probably among the first things to go. All bits and bytes matter here, so to speak. So maybe this sparse file attribute rarely gets to do something. I don't know…
There doesn't seem to be an easy way to disable this behavior in qBittorrent. And the concept (inherent to NTFS) is maybe not bad to just have anyways. Why write something you don't have to, until you actually need to? Just unnecessary wear on the drive. Although practically you're not going to wear out your SSD before it becomes obsolete anyway, even if you do download some large .iso:s now and then.
You CAN enable preallocation though where it starts by writing zeroes to the whole area where the file is going to be. This disables sparse files also. This could be somewhat useful for hard drives to minimize fragmentation, but is completely irrelevant for SSDs, which purposely fragment files for both parallelized performance gains and wear leveling reasons anyway. So if you preallocate it's not going write the data to the SSD any differently than how it would have otherwise. You're just stuck waiting for it to write the empty file first. In fact, it's gonna mean more wear, since it first writes zeroes and then replaces some of them with ones.
On the otherhand, if I need to clear the sparse file attribute afterwards, then I'm going to have to write the whole file again anyway… But probably not worth enabling, just because of the occasional release that doesn't work outright in pyReScene.
But this whole thing got me thinking!
I also got this release through qBittorrent and the .iso also had 0x220, but could automatically be recreated with pyReScene.
https://www.srrdb.com/release/details/A.Plague.Tale.Innocence-CODEX
I checked the details of the .srr and, sure enough, ATTR is supposed to be 0x20. So how come ATTR didn't matter for this release?
Block: RAR File; offset: 0x3086 (12422 bytes)
|Header bytes: 79e374028149000fb2e60e0080899e022afcd52b2195ae4e1d312100200000000000000009000000636f6465782d612e706c616775652e74616c652e696e6e6f63656e63652e69736f
|HEAD_CRC: 0xE379
|HEAD_TYPE: 0x74 (RAR File)
|HEAD_FLAGS: 0x8102
| 0x8000 LONG_BLOCK (ADD_SIZE field present)
| 0x0002 LHD_SPLIT_AFTER (file continued in next volume)
| 0x0100 LHD_LARGE (only used for files larger then 2GiB)
| 0x0000 LHD_WINDOW (Dictionary size 64 KiB)
|HEAD_SIZE: 0x49 (73 bytes)
+PACK_SIZE: 249999887 bytes (ADD_SIZE + HIGH_PACK_SIZE field)
+UNP_SIZE: 41314516992 bytes
+HOST_OS: Windows used to create this file block.
+FILE_CRC: 2BD5FC2A
+FTIME: 2019-05-14 18:41:02
+UNP_VER: Version 2.9 is needed to extract.
+METHOD: Fastest compression
+NAME_SIZE: always present
+ATTR: 20
+HIGH_PACK_SIZE: 0
+HIGH_UNP_SIZE: 9
+FILE_NAME: codex-a.plague.tale.innocence.isoAnd furthermore, I didn't have to mess around with any dates on the .iso file either. This release does not have LHD_EXTTIME in the header and therefore not that block at the bottom.
+modification time: 2005-05-13 18:57:29
+creation time: UNKNOWN
+last access time: UNKNOWN
+archival time: UNKNOWNSo then I did some testing:
1.
I copied the first release again, so I got a new create date, but the old modified date and no sparse file attribute and ran the command again and the volumes still had the correct checksums!
2.
So I changed the modified date to something modern as well and reran, but then I got the wrong checksums…
Putting this together, it seems that FTIME does not matter, since
+creation time: UNKNOWN… and "UNKNOWN" might be null or something in the actual .srr file. Might just be presented as "UNKNOWN" when you use —details.
But modification time is there, so that needs to match. The CODEX release .srr had none of that stuff, so I guess they did not use to store the .iso file modification time in their .rar files, whereas Razor 1911 do? And that's why those lines appear in the .srr file maybe? Maybe if even only one of them is there in the archive, then all four lines are included in the .srr or something?
It seems the default behavior in WinRAR is "-tsm4 -tsc0 -tsa0", which means full NTFS precision on the modification time and the other ones are omitted. I.e. if you don't specify anything this is what you get. Maybe CODEX went out of their way to not include anything.
Aaanyway, back to why having the wrong attributes on the file still worked for the second release here. My theory is that when you use pyReScene to automatically try to recreate the files, then this
Grabbing large enough data piece size for testing.when it creates its sample, it inadvertently also loses the sparse file attribute on its sample copy. So the resulting checksums check out and it starts the process. I then also think that WinRAR - at least when compressing data - does not (or can't) consider sparse files, so the sparsity is then again lost in the actual compression process. I even tried to extract the second release from the files I had generated before and the resulting .iso was not sparse, even though the .iso I had used to create the rars was.
I checked some other recent Razer 1911 .srr files and it seems for at least AAA titles that this is usually there.
+modification time: 2005-05-13 18:57:29So in summary if the automatic recreation doesn't work for future Razer 1911 then:
0. Check the details of the .srr file. Perhaps not needed, since the next steps will probably work.
srr.exe --details name.srr > name.txt1. Copy the file to get rid of the sparse file attribute.
2. Set this modification date in PowerShell.
(Get-Item "name.iso").LastWriteTime=("13 May 2005 18:57:29")3. Manually create the rar files, of course paying attention to the volume size.
2015-11-18_rar530.exe a -m1 -mdG -v250000000b -s- -ds -mt12 -vn -o+ -ep -idcd -ma4 name.rar name.isoGlad to hear you got it working! The sparse thing was new to me, so that was interesting to learn about :)
I also got this release through qBittorrent and the .iso also had 0x220, but could automatically be recreated with pyReScene.
https://www.srrdb.com/release/details/A.Plague.Tale.Innocence-CODEX
I checked the details of the .srr and, sure enough, ATTR is supposed to be 0x20. So how come ATTR didn't matter for this release?
The reason for this is that pyReScene works differently, it only checks for the actual compressed data to match, and doesn't need the rest of the RAR file (headers) to still be able to recreate the release. How this works is that it goes through all the settings, compressing the sample and checking the FILE_CRC header to match. Once it finds a match it will compress the (whole) file and then perform a 'merge' of the .srr and .rar files by copying the header information from the .srr and the compressed data from the .rar. You can see it create these intermediate rar files in the temporary folder.
Because it doesn't use the headers from the .rar it doesn't matter that the ATTR is wrong or the timestamp is wrong. It only matters in edge cases like this, where the result of compression (apparently) depends on the exact size of the data that is getting compressed. In such cases pyReScene will never find a "good" rar version. In theory it should be so that if you set the correct timestamps and ATTR and force the correct volume size in main.py, pyReScene should be able to find a "good" rar and reconstruct the release. A specific volume size can be forced by replacing the self.split by a hard coded volume size like "-v250000000b" in function arglist().
So then I did some testing:
…
Putting this together, it seems that FTIME does not matter, since
Interesting, I had a look around and it seems that FTIME is indeed not the creation time, but (also) the modification time. You can see this from the code that is reading the headers. First the 4 bytes after the checksum are read into a variable FileTime (this corresponds to the FTIME field):
And then later just before it parses the LHD_EXTTIME contents, it sets the mtime on the header, which is then further updated with data from the extended timestamp inside the if() statement:
hd->mtime.SetDos(FileTime);
if ((hd->Flags & LHD_EXTTIME)!=0)
So I think what you mean is that the creation time does not matter, it is only the modification time that matters.
I then also think that WinRAR - at least when compressing data - does not (or can't) consider sparse files, so the sparsity is then again lost in the actual compression process. I even tried to extract the second release from the files I had generated before and the resulting .iso was not sparse, even though the .iso I had used to create the rars was.
This bit I don't quite follow. You mean that you compressed a sparse file, but when you extracted it was no longer sparse?
I checked some other recent Razer 1911 .srr files and it seems for at least AAA titles that this is usually there.
+modification time: 2005-05-13 18:57:29
That is a neat Easter egg to find! I wonder what the significance of this date is …
Although some releases do have different (but still incorrect) dates, like Vampire_The_Masquerade_Swansong-Razor1911 which is dated 2010-05-12 16:02:33.
Oh alright. Then I understand better how it works. Forget about my assumptions. :)
What checksum is that on FILE_CRC? Because it's not the checksum of the first .rar volume.
I normally just use the compiled executables, but I do have the Python version set up as well for use with pyAutoReScene. So I tried to modify main.py for that, but I'm not sure I'm doing it right, because it keeps spitting out temp files of those 180MB or so. Shouldn't they be 250MB is the thinking? So that it can compare one of those with the correct size to FILE_CRC?
I commented out row 1923 and added the row below that.
-
So I think what you mean is that the creation time does not matter,
Yes!
This bit I don't quite follow. You mean that you compressed a sparse file, but when you extracted it was no longer sparse?
It doesn't matter, but, well, yes. For the second release it worked fine, even with 0x220, so it was sparse. But as you explained, when using pyReScene, it copies the header data from the .srr file, so it doesn't matter what attributes my .iso file had. My bad attributes don't go into the archive.
Haven't found a significance to that date either. Perhaps different packers in the group just use different dates over and over. :) I'm going to try 2005 first next time though.
What checksum is that on FILE_CRC? Because it's not the checksum of the first .rar volume.
It's the checksum over the compressed data stored in that RAR File block. Except for the last volume, where it has a FILE_CRC that matches the CRC of the whole file. You can see this in the comments of function try_rar_executable().
I commented out row 1923 and added the row below that.
Your change looks good (the expected size is ~238MiB). This works for me, I can see it create temporary files with the size I set here :S
What is the size of the sample? Maybe the sample is too small or compresses too well? Although the sample-size detection should take care of that :/
I'm not sure I'm following regarding FILE_CRC. Why isn't it the same as the checksums in the .sfv file? Especially if we're trying to force it to use 250000000 as the size in the testing phase, which is the same as what the end rar volumes are supposed to have.
You are talking about the pyReScene_compressed.rXX files, right? They all get this size for me still:
Size: 180 MB (189 417 906 bytes)
Size on disk: 180 MB (189 419 520 bytes)I'm guessing the slight difference is due to the "sectors" of the SSD or something, like how much data must be written at minimum at a time. Not due to the formatting, but how the NAND cells work, but not sure.
When you say sample here, do you mean pyReScene_data_piece.iso I see in the temp folder?
And I'm still trying it out with this release, so it's a 50GB file. :)
https://www.srrdb.com/release/details/A_Plague_Tale_Requiem_v1.3.0.0-Razor1911
I'm not sure I'm following regarding FILE_CRC. Why isn't it the same as the checksums in the .sfv file? Especially if we're trying to force it to use 250000000 as the size in the testing phase, which is the same as what the end rar volumes are supposed to have.
Suppose you have a release with 3 volumes:
You are talking about the pyReScene_compressed.rXX files, right? They all get this size for me still:
Yeah, with the change above these files change size according to what I put in main.py. I don't know why it's not working for you, maybe try to make some change that is easier to check and see if that gets applied (i.e. are you really running the modified code).
When you say sample here, do you mean pyReScene_data_piece.iso I see in the temp folder?
Yes
I've given it a try too with pyReScene, but I can't get it to detect the correct RAR version. I've tried various values for -v…b until I found one that results in the right value for the PACK_SIZE field1. With this the right amount of data should be in the first volume and the split should happen at the right place, but still it doesn't result in a good checksum when pyReScene runs RAR on the sample file. I don't have a good explanation for why this is. Maybe the actual size of the file triggers some special condition?
1. My value was -v249999977b, but it will be different for you because I have other mods that e.g. change the name of the sample file. For you it may be -v249999989b, but ymmv. The difference is that I use a different (shorter) name for the sample. To check this I stopped pyReScene once it gets "stuck" (trying the full-file compression) and looked at the pyReScene_compressed.rar in a hexeditor.
I think I'm quickly reaching my limits to fully understand this and to keep testing it. Not sure how to verify that it reaches that code. I think I'm just going to be happy with the manual workaround for Razer 1911 that you showed me. :) And be sad that FairLight uses macOS. :(
I do have access to an old Macbook Air though, so I decided to do some testing with it. I managed to scrounge up no less that 29 versions of RAR for macOS. Missing some of the really old ones and I don't have any beta versions. Not even sure it's all of them, but I have from 3.9.3 to 6.20.
I soon realized though that most of them wouldn't run on the system, since they are 32-bit and Apple dropped support for that in Catalina and onwards in 2019. Around that time I think all future macOS versions of RAR also became 64-bit. So unless the good people at FairLight are running outdated operating systems, I'm assuming they have to be using a fairly recent version of RAR, which narrows it down a lot.
That computer is really slow though, so I've only done one test and it didn't pan out. But I'm not even sure I'm doing it correctly. I'm running the same, custom command as the one above (but for an FLT release of course). Compressing the whole release and testing the .sfv file. Perhaps I'll try a few more though and see if I get lucky. And perhaps I should do it on something smaller than this. :D
https://www.srrdb.com/release/details/UNCHARTED_Legacy_of_Thieves_Collection-FLT
About FILE_CRC though, I think I get it! It's the CRC of the compressed payload of the volume. That would be what you've highlighted in blue above I take it? I.e. the header/meta data of the volume itself is not included. That's in the .sfv though, the CRC of the full file, header data and all, which is generated afterwards by a separate process. FILE_CRC is generated and stored in the header during the compression process.
So it compresses "some" data and calculates CRC for that. Then stores that in the header of that volume and moves on to the next volume and caculates that, and so on. The exception being the last volume that has the CRC of the whole compressed file, instead of the CRC of the compressed remaining portion, as you explained.
(I wonder how it does that though, the last bit. If it can sort of "add up" the partial checksums and know what the last one will be immediately when it reaches there. Or if it's maintaining an ongoing, parallel checksum calculation for the whole file as it is progressing. Because otherwise it'd have to recalculate the whole thing when it's done, to be able to close out/finalize the archive properly, which takes time. Anyway, that doesn't matter. Just curious.)
But what I'm not really understanding here is this. If the resulting volume (header and all) should result in EXACTLY 250000000, and since we're dealing with compression here, it cannot know beforehand how much uncompressed data it will be able to put in the first volume, because it can't know exactly how well it will compress. So it has to compress more, too much if you will, and then cut it off at 250000000. Well, slightly less than that, because it needs to fit the header data in there as well, so that the result comes out to exactly 250000000. It can know how large the header is though.
And if the size of what you are compressing affects how it compresses the data, i.e. "later" data can influence how it chooses to compress "earlier" data, then how is it going to be able to determine enough of a sample size so that it's going to leave the "earlierly" (is that a word?) compressed data alone? So it will match what it would do if you just let it compress the whole thing?
To put it another way, 250MB of compressed data into one volume might look different from the first volume of two of 500MB compressed data, since it has more to look at during the compresseion and might make different decisions. Despite the first 250MB of uncompressed data of the 500MB file being identical in this example.
As I'm typing this I realize this is not a problem for when you're generating the original archive. You're just compressing data in one end and out the other comes a stream of compressed data. Whatever it decides to write during this process is up to the algorithm. How far it decides to "look ahead" and when it actually writes something doesn't matter, but WHEN it does you can be sure that that's not going to change anymore during the compression process. The algorithm is satisfied with that. So then when it's output enough data for a volume (minus the header), it can just chop that off, calculate FILE_CRC, and write a file. And then go back to waiting for more data to arrive.
So if I specify "volumes of 250000000, please", how much data is pyReScene actually grabbing to test the compression so that it arrives at slightly less that 250000000 of compressed data? I guess if the sample size is sufficiently larger, it will compress similarly enough, as if you'd compress the whole thing, to not affect the outcome for the first volume.
Alright, so for posterity, I tried out all the RAR versions for macOS (that I could find), starting with the most recent one and going backwards. And testing the CRCs after a few files had been generated during each attempt.
Used this release: https://www.srrdb.com/release/details/UNCHARTED_Legacy_of_Thieves_Collection-FLT
This is the command I used. I guess "-mt12" could have been lower. That poor dual core from 2013…
./rar a -m1 -mdG -v250000000b -s- -ds -mt12 -vn -o+ -ep -idcd -ma4 name.rar name.isoRan these versions:
| Version | Result |
|---|---|
| rarmacos-x64-620.tar.gz | Bad CRC |
| rarmacos-x64-612.tar.gz | Bad CRC |
| rarmacos-x64-611.tar.gz | Bad CRC |
| rarmacos-x64-610.tar.gz | Bad CRC |
| rarosx-6.0.2.tar.gz | Bad CRC |
| rarosx-6.0.1.tar.gz | Bad CRC |
| rarosx-6.0.0.tar.gz | Bad CRC |
| rarosx-5.9.1.tar.gz | Bad CRC |
| rarosx-5.9.0.tar.gz | Bad CRC |
| rarosx-5.8.0.tar.gz | Bad CRC |
| rarosx-5.7.1.tar.gz | Bad CRC |
| rarosx-5.7.0.tar.gz | Bad CRC |
| rarosx-5.6.1.tar.gz | Bad CRC |
| rarosx-5.6.0.tar.gz | Bad CRC |
| rarosx-5.5.0.tar.gz | Didn't run (32-bit) |
Like I said, I might be missing some smaller versions in between, including beta versions, but I don't think this release requires some weird date modifications when looking closer att the .srr file. But I'm of course not sure about that.
So maybe I was just unlucky that this didn't work. Or there's more to it to replicate this on macOS.