Glacie 0.1.0: library for manipulation game files

[lixiss]

I had figured that you're supposed to use command line, but then what are you supposed to type in ?

I'm not sure what I'm understand you correctly. (English is not my native language.)
Does you ask for suggestions from where or how to call tools? Answer might be varied if you already familiar with basic OS (operation system) concepts or no.
For anything what i'm call multiple times, I'm usually create .cmd file. Otherwise i'm just type commands in terminal (also known on windows as command prompt).

If you use the portable version glacie-cli-0.3.1-alpha.2-win-x64, can you put it wherever you want ? Then using only cmd.exe is enough ?

Yes, you can put it where you want, and it doesn't require any other dependencies to be installed.
However there is minor therminology issue - there is not "portable" in usual sense - this bundle built for win-x64 arch and will work only on Win7+ x64,
but generally this project can be runned on linux (but i'm doesn't very care about).

You said you wrote this to generate a report :
gx validate 01-tqae.gxproject --output-html-report=validation-report-0.4-3-tqae-2.9-simple-1.html

What is gx ?   I see gx-arc.exe and gx-arz.exe, no gx.exe. Is this normal ? Or should the correct command be :
gx.arz validate 01-tqae.gxproject --output-html-report=validation-report-0.4-3-tqae-2.9-simple-1.html

What's correct: `gx` is another command line tool, but from current (development) version, and i'm doesn't made pre-built executables for current version.
However their sources are published, it under development, under heavy cleanup and refactoring, and has enough already known issues.
I'm doesn't publish because in current state practically it is not very usable (in sense what if you enable deeper validation it will yield ton of errors,
and at least half of them should not be reported once features will be done (they are technically still errors, but there is no sense report if record is not used at all,
and this issue heavily limit usage). So, i'm just post current state / direction.
If you want:
1) i'm can prebuilt it.
2) You can built it alone, all what you need is install .NET Core SDK 3.1 (currently is 3.1, after 5.0 release it will be .NET 5.0), clone source code repository,
and build it by calling dotnet build.
If you want to try, i'm may create additional instructions. (Eventually they need anyway in readme or wiki which empty currently.)

What is 01-tqae.gxproject ? What is this file format ? How to create a .gxproject file from database.arz ? Where do we put the .gxproject file ? Because the command doesn't specify the path, should we just put the .gxproject in ...\glacie-cli-0.3.1-alpha.2-win-x64 ?

.gxproject - currently just simple xml file with next content (G:\Games\TQAE - is TQAE install location):

Code: [Select]

<gx-project>
  <source path="G:\Games\TQAE" />
  <metadata path="G:\Games\TQAE" />
</gx-project>

Or for GD:

Code: [Select]

<gx-project>
  <source path="Z:\Games\Grim Dawn 1.1.7.1 (39085)" />
  <source path="Z:\Games\Grim Dawn 1.1.7.1 (39085)\gdx1" />
  <source path="Z:\Games\Grim Dawn 1.1.7.1 (39085)\gdx2" />
  <metadata path="Z:\Games\Grim Dawn 1.1.7.1 (39085)" />
</gx-project>

It also incomplete, there is need at least "engine" option, may be some other.
Generally project describes subject of work.

Source defines source module. "Module" generally is combination of database (.arz) and resources.
Source is source, because it is supposed to consume/read only, while target is something what supposed we want to build.
However, target you can ignore, as it currently not supported (and as validation doesn't make any writes - it doesn't required.)

Metadata specifies where to search metadata files (in mine internal format), or where to search template files.
If game folder is specified, then it load them from Toolset\Templates.arc or database/templates.arc, but generally can be any folder with .tpl files
or path to .arc or .zip file.

Also SV- templates currently it will not be able to load, because if i'm not mistaken, Actor2.tpl? redefine same field multiple times.
I'm probably improve reader to eat this case automatically.

So... patience... work in progress.

--output-html-report=validation-report-0.4-3-tqae-2.9-simple-1.html
So for that part here, I suppose it's just         --output-html-report=[custom_report_name.html]. Seems straightforward.

Yes, it is just output file name. Tool already write in terminal/console lot of messages and there is not very easy to read them if their number more than ~10.
HTML designed very fast and ugly.

[soa]

I'm not sure what I'm understand you correctly. (English is not my native language.)
Does you ask for suggestions from where or how to call tools? Answer might be varied if you already familiar with basic OS (operation system) concepts or no.
For anything what i'm call multiple times, I'm usually create .cmd file. Otherwise i'm just type commands in terminal (also known on windows as command prompt).

I think that a lot of the forum readers, myself included, rarely or never use command line tools and have limited knowledge in programmation.
I sometimes use command line, sometimes do a piece of code, but it's very limited and I never use dotnet framework, javascript, c#.
What I was asking is very simple, is it enough to use only cmd.exe (command prompt) in windows to make your tools work (I mean, when it's built, not the current version of gx) ?
I just wanted to be sure it wasn't needed to use any other more complex tool to execute the code.

What's correct: `gx` is another command line tool, but from current (development) version, and i'm doesn't made pre-built executables for current version.
However their sources are published, it under development, under heavy cleanup and refactoring, and has enough already known issues.
I'm doesn't publish because in current state practically it is not very usable (in sense what if you enable deeper validation it will yield ton of errors,
and at least half of them should not be reported once features will be done (they are technically still errors, but there is no sense report if record is not used at all,
and this issue heavily limit usage). So, i'm just post current state / direction.
If you want:
1) i'm can prebuilt it.
2) You can built it alone, all what you need is install .NET Core SDK 3.1 (currently is 3.1, after 5.0 release it will be .NET 5.0), clone source code repository,
and build it by calling dotnet build.
If you want to try, i'm may create additional instructions. (Eventually they need anyway in readme or wiki which empty currently.)

OK, I prefer to wait until there is a usable version, even if it is limited to a few features.

.gxproject - currently just simple xml file with next content (G:\Games\TQAE - is TQAE install location):

Code: [Select]

<gx-project>
  <source path="G:\Games\TQAE" />
  <metadata path="G:\Games\TQAE" />
</gx-project>
It also incomplete, there is need at least "engine" option, may be some other.
Generally project describes subject of work.

Source defines source module. "Module" generally is combination of database (.arz) and resources.
Source is source, because it is supposed to consume/read only, while target is something what supposed we want to build.
However, target you can ignore, as it currently not supported (and as validation doesn't make any writes - it doesn't required.)

Metadata specifies where to search metadata files (in mine internal format), or where to search template files.
If game folder is specified, then it load them from Toolset\Templates.arc or database/templates.arc, but generally can be any folder with .tpl files
or path to .arc or .zip file.

So, If I put something like this, it should work, as long as there is a database folder in C:\My Games\Titan Quest - Immortal Throne\custommaps\sv_aer, with a .arz file in it  ?

Code: [Select]

<gx-project>
  <source path="C:\My Games\Titan Quest - Immortal Throne\custommaps\sv_aer" />
  <metadata path="C:\My Games\Titan Quest - Immortal Throne\Working\Database\Templates" />
</gx-project>


Also SV- templates currently it will not be able to load, because if i'm not mistaken, Actor2.tpl? redefine same field multiple times.
I'm probably improve reader to eat this case automatically.

I haven't created those templates, most are not even used in the mod, like Actor2.tpl, so even if they don't work it's not a big deal.
I think they were mostly experiments and haven't been cleaned up when the experiment failed.
So, I'm gonna delete them and only keep a few that work.

[lixiss]

I think that a lot of the forum readers, myself included, rarely or never use command line tools and have limited knowledge in programmation.
I sometimes use command line, sometimes do a piece of code, but it's very limited and I never use dotnet framework, javascript, c#.
What I was asking is very simple, is it enough to use only cmd.exe (command prompt) in windows to make your tools work (I mean, when it's built, not the current version of gx) ?
I just wanted to be sure it wasn't needed to use any other more complex tool to execute the code.

I'm understand... Clear. Surely it is enough cmd.exe to run gx/gx-arc/gx-arz programs.

So, If I put something like this, it should work, as long as there is a database folder in C:\My Games\Titan Quest - Immortal Throne\custommaps\sv_aer, with a .arz file in it  ?

Code: [Select]

<gx-project>
  <source path="C:\My Games\Titan Quest - Immortal Throne\custommaps\sv_aer" />
  <metadata path="C:\My Games\Titan Quest - Immortal Throne\Working\Database\Templates" />
</gx-project>


Absolutely correct.

Also SV- templates currently it will not be able to load, because if i'm not mistaken, Actor2.tpl? redefine same field multiple times.
I'm probably improve reader to eat this case automatically.

I haven't created those templates, most are not even used in the mod, like Actor2.tpl, so even if they don't work it's not a big deal.
I think they were mostly experiments and haven't been cleaned up when the experiment failed.
So, I'm gonna delete them and only keep a few that work.

Well, it is probably not Actor2... mmmm. Forget. Anyway, it is not important now actually, don't worry about. Errors/trash are exist even in templates bundled with game, all templates "Copy of..." are trash, as easy example. This becomes issue technically, especially in GD, because they can't be legally parsed, because some of them include non-existent templates. I has special exclude list for them. There is just "mine" technical detail and how i'm treat "metadata". For validation / normal consumption they are internally built to self-sufficient data structure, but collected / parsed / processed all-at-once, because some information can be built only on complete graph. However, gx-arz and ArtManager only pull specified by record template one by one when process record. This is why gx-arz and ArtManager "doesn't care" about trash templates which lives with game forever. But this aspect completely manageable.

[lixiss]

OK, I prefer to wait until there is a usable version, even if it is limited to a few features.

BTW, i'm not sure if you was patient enough to read all topic from start, but I'm time-to-time use SV-AERA database to tests, mostly because it one of biggest in mine collection. Executing gx-arz/gx-arc optimize/repack commands can save some bits, generally for TQAE-favor files this is makes ~5% smaller sizes of .arz or .arc files. 5% is not impressive per se, but in absolute cumulative numbers is might be about 30-40mb for .arz + .arc. I'm doesn't sure what this, as self-goal, worth of efforts (because relative save in modern world is pathetic), but there is still something, and there is really easy commands to start with.

In practice, if you unpack prebuilt glacie-cli-0.3.1-alpha.2-win-x64.7z somewhere, let it be w:\gx\bin-0.3.1 (you can make it simplier, it is mine real path).
Then you put your's database.arz in w:\somewhere, then in command prompt you can do:

Code: [Select]

C:\>cd /d w:\somewhere

w:\somewhere>w:\gx\bin-0.3.1\gx-arz optimize --repack database.arz
Optimized: DBR file references
  # of Remapped Strings: 40,458
  Estimated Size Reduction: 2,722,296 bytes
Optimized: TPL file references
  # of Remapped Strings: 737
  Estimated Size Reduction: -108 bytes          // LOL! Just noticed, what calculation is invalid or... don't know. This value should not be negative.
Written: database.arz
Optimized:
   Input Database Size: 77,618,799 bytes
  Output Database Size: 73,685,079 bytes (94.9%)

w:\somewhere>

cd /d - cd with /d switch changes current directory AND volume (e.g. disk). cd without /d will keep you in C: disk (but will switch current directory on w: volume). So, most of time if you ever use cd - always specify /d.

--repack switch actually enables all implemented "optimizations". This normalizes all .dbr file references to the form how they stored in .arz file, for example. Simplier: "records\some.dbr" and "Records/Some.dbr" references same entity, but there is two different strings which are stored. However, biggest benefit, comes from repacking with highest compression level, and second benefit, implemented as "state of art" is to rearrange (sort by some rule) string table to provide better savings across all tested databases (basically TQAE, SV-AERA and GD).

Everything below is just play with things deeper.

NOTE: Optimizations doesn't do destructive changes, nor do logical changes, but this is still changes over existing data. So DOES NOT execute this over game files or files which you want to keep in original state. Always keep backups or anything.

Another example (assuming you have original database.arz like in example before):

Code: [Select]

w:\somewhere>w:\gx\bin-0.3.1\gx-arz optimize -Oostrtable -Orecompress database.arz
Written: database.arz
Optimized:
   Input Database Size: 77,618,799 bytes
  Output Database Size: 76,689,483 bytes (98.8%)

Command above attempts (speculative attempt) to rearrange string table to most profitable way and recompress database. -Oostrtable gives reproductible string table layout, which sometimes important.

Let's try do same but without this switch, but always will build new string table and emit arz file (-Orstrtable is different from -Oostrtable what string table is regenerated (this allow remove unused entries if they exist), and with enabled multi-core/multi-processor support (enabled by default but may be switched off) - leads what after every run due to natural encoding race, order in which records appear in file are shuffled. There is not a problem from correctness, just affect encoding and compression effectiveness, because every next time you do same things in example below - you actually recompress different sequence of data). Again, starting from unmodified database.arz:

Code: [Select]

w:\somewhere>w:\gx\bin-0.3.1\gx-arz optimize -Orstrtable -Orecompress database.arz
Written: database.arz
Optimized:
   Input Database Size: 77,618,799 bytes
  Output Database Size: 76,252,835 bytes (98.2%)  // WHOOP! I'm feel lucky.

w:\somewhere>w:\gx\bin-0.3.1\gx-arz optimize -Orstrtable -Orecompress database.arz
Written: database.arz
Optimized:
   Input Database Size: 76,252,835 bytes
  Output Database Size: 76,178,366 bytes (99.9%)  // WHOOP! Even more lucky!

w:\somewhere>w:\gx\bin-0.3.1\gx-arz optimize -Orstrtable -Orecompress database.arz
Written: database.arz
Optimized:
   Input Database Size: 76,178,366 bytes
  Output Database Size: 76,400,801 bytes (100.3%)  // AARGH! Not lucky!

w:\somewhere>w:\gx\bin-0.3.1\gx-arz optimize -Orstrtable -Orecompress database.arz
Written: database.arz
Optimized:
   Input Database Size: 76,400,801 bytes
  Output Database Size: 76,268,875 bytes (99.8%)   // :) Lucky again!

w:\somewhere>w:\gx\bin-0.3.1\gx-arz optimize -Orstrtable -Orecompress database.arz
Written: database.arz
Optimized:
   Input Database Size: 76,268,875 bytes
  Output Database Size: 76,894,228 bytes (100.8%)  // WHAAT?

// Let's back to -Oostrtable strategy: it is worse than best possible seen, but it is stable.

w:\somewhere>w:\gx\bin-0.3.1\gx-arz optimize -Oostrtable -Orecompress database.arz
Written: database.arz
Optimized:
   Input Database Size: 76,894,228 bytes
  Output Database Size: 76,689,483 bytes (99.7%)

w:\somewhere>w:\gx\bin-0.3.1\gx-arz optimize -Oostrtable -Orecompress database.arz
Written: database.arz
Optimized:
   Input Database Size: 76,689,483 bytes
  Output Database Size: 76,689,483 bytes (100.0%)  // It is just stable (and on average it gives good results.)

There is just can be treated as "training session" for CLI experience for anyone who want.