Some of the XML files are huge, Notepad++ don't cut it.

Please post here for questions and discussion about data, event, art and sound modding and the game editor for Distant Worlds.

Moderator: MOD_DW2

Post Reply
Foraven
Posts: 319
Joined: Mon Apr 20, 2009 1:32 am

Some of the XML files are huge, Notepad++ don't cut it.

Post by Foraven »

Alright, I want to make mass changes in the shiphull file, but doing so would take me ages if I did it one by one (there are thousands of entries). I did try to use Excel for this, and it does make changing stuff extremely easy but I can't export it back in the original XML format (I only have Excel 2007). Is there any other way to make changes in batch so I can get things done without spending days manually tweaking the file?

Edit: I started to use the Distant Worlds 2 Mod Maker and it does make making mods easier. However, doing things like changing say all engine max size to 25 would not be easy as I would have to look at every engine entry. I did start my mod with Excel earlier and it was very easy to make that kind of changes using the filter options; knowing how to properly export it back to XML would be my ideal solution.
Buio
Posts: 341
Joined: Wed Nov 21, 2012 7:40 pm

Re: Some of the XML files are huge, Notepad++ don't cut it.

Post by Buio »

I don't use Notepad++, but try turning of stuff like highlightning, wrapping etc. Anything that need to dynamically change the file for presentation on the screen. Because on huge files these require lot of resources.

When opening "ShipHulls.xml" in VSCode I get a popup saying these were turned off per default to avoid issues (you can force it but risk instability).

Or try VSCode yourself, it is free.
https://code.visualstudio.com/
Foraven
Posts: 319
Joined: Mon Apr 20, 2009 1:32 am

Re: Some of the XML files are huge, Notepad++ don't cut it.

Post by Foraven »

Buio wrote: Sun Aug 28, 2022 6:19 am I don't use Notepad++, but try turning of stuff like highlightning, wrapping etc. Anything that need to dynamically change the file for presentation on the screen. Because on huge files these require lot of resources.

When opening "ShipHulls.xml" in VSCode I get a popup saying these were turned off per default to avoid issues (you can force it but risk instability).

Or try VSCode yourself, it is free.
https://code.visualstudio.com/
I don't have much problem to open huge files with Notepad++, my issue is I don't have the proper tools to do the kind of modding I want to do. I don't want to change values 1 at a time, I want to be able to do it for multiple entries at once without changing the wrong values along. Usually I can do this in Notepad++, but XML entries are set in a way that don't allow me to use the tools I have for it.
Buio
Posts: 341
Joined: Wed Nov 21, 2012 7:40 pm

Re: Some of the XML files are huge, Notepad++ don't cut it.

Post by Buio »

Aha, you want to edit the data easily as a database or spreadsheet. Don't know about that for XML. Had it been JSON format it's easier to find free alternatives for.
User avatar
frankycl
Posts: 401
Joined: Mon Mar 01, 2021 5:16 pm

Re: Some of the XML files are huge, Notepad++ don't cut it.

Post by frankycl »

Foraven wrote: Sun Aug 28, 2022 12:52 pm I don't have much problem to open huge files with Notepad++, my issue is I don't have the proper tools to do the kind of modding I want to do. (...)
Why don't you use regular expressions ? : https://regexr.com/ - They are easy to use with any (WIN-) text-editor and with Notepad++, too. ;)

And bsds: Most values are easy to edit with mere "search and replace" in a breeze - you just have to add some of the tags to your search-values, to make them a specific search. ;)
(What values / types do you want to edit exactly ? - If you give us a bit more details we might be able to help you even further. ;) )
Foraven
Posts: 319
Joined: Mon Apr 20, 2009 1:32 am

Re: Some of the XML files are huge, Notepad++ don't cut it.

Post by Foraven »

frankycl wrote: Sun Aug 28, 2022 6:43 pm
Foraven wrote: Sun Aug 28, 2022 12:52 pm I don't have much problem to open huge files with Notepad++, my issue is I don't have the proper tools to do the kind of modding I want to do. (...)
Why don't you use regular expressions ? : https://regexr.com/ - They are easy to use with any (WIN-) text-editor and with Notepad++, too. ;)

And bsds: Most values are easy to edit with mere "search and replace" in a breeze - you just have to add some of the tags to your search-values, to make them a specific search. ;)
(What values / types do you want to edit exactly ? - If you give us a bit more details we might be able to help you even further. ;) )
I would need more explanations because I don't quite understand what you mean there (or what is on the site you linked). For sure the search and replace included in Notepad++ don't work because it only scan one line at a time, I can't include what was in the line before (or after) to make sure I am editing the right stuff. If your utility can do that, great but I have no idea how to use it (or even if I could use it on a huge xml file).
User avatar
frankycl
Posts: 401
Joined: Mon Mar 01, 2021 5:16 pm

Re: Some of the XML files are huge, Notepad++ don't cut it.

Post by frankycl »

Foraven wrote: Tue Aug 30, 2022 10:28 pm
frankycl wrote: Sun Aug 28, 2022 6:43 pm Why don't you use regular expressions ? (...)
I would need more explanations because I don't quite understand what you mean there (or what is on the site you linked). For sure the search and replace included in Notepad++ don't work because it only scan one line at a time, I can't include what was in the line before (or after) to make sure I am editing the right stuff. If your utility can do that, great but I have no idea how to use it (or even if I could use it on a huge xml file).
Ok, first: regular expressions are no "utility", but codes that you can use in most text-editors, like e.g. Notepad++, to do things like "search for all text/numbers/symbols at the beginning/ending of a line" or "...after/before (defined text/numbers/symbols)", etc. - therefore you can do very specific searches and replace the found text/numbers/symbols with whatever you like - for euch find different or for all finds altogether for the whole file.
And in order to use this you must click the circled checkbox for "regular expressions" in the search/replace dialogue from Notepad++ (or whatever text-editor you use). ;)
(and on the site I linked above you can find the above mentioned codes (and their explanations) you need to insert in the search/replace dialogue(s) I mentioned)

BUT - first of all you need to know what you are doing, i.e. how those files/entries you want to edit are used in the game - and what you writes indicates that this is not the case / you need to get some additional knowledge of these things. I would recommend to try the following thread and the attached files for this: https://www.matrixgames.com/forums/view ... 4d0f26ec0c ;)

However the easiest thing would probably be that you tell us what you want to edit exactly in the files (preferably in code here in the forum !) - and then we could see if we/I can help you further (like e.g. with the needed regular expressions from above; - I'm no expert with those things myself, but I know somebody who is - and who most likely would help us. :D ). ;)
Foraven
Posts: 319
Joined: Mon Apr 20, 2009 1:32 am

Re: Some of the XML files are huge, Notepad++ don't cut it.

Post by Foraven »

frankycl wrote: Wed Aug 31, 2022 11:06 am However the easiest thing would probably be that you tell us what you want to edit exactly in the files (preferably in code here in the forum !) - and then we could see if we/I can help you further (like e.g. with the needed regular expressions from above; - I'm no expert with those things myself, but I know somebody who is - and who most likely would help us. :D ). ;)
Well, it is simple...

Code: Select all

			<ComponentBay>
				<ComponentBayId>2</ComponentBayId>
				<Type>Engine</Type>
				<MaximumComponentSize>50</MaximumComponentSize>
				<MeshName>#engine1</MeshName>
				<RotationHalfArcRange>0</RotationHalfArcRange>
				<DisplayEffectRescaleFactor>
					<X>0.75</X>
					<Y>0.75</Y>
					<Z>1</Z>
				</DisplayEffectRescaleFactor>
				<DisplayEffectOffset>
					<X>0</X>
					<Y>0</Y>
					<Z>0</Z>
				</DisplayEffectOffset>
			</ComponentBay>
If I want to change all engine related MaximumComponentSize to 25, I can't use the replace function in Notepad ++. I can change them one by one, finding the <Type>Engine</Type> then changing each Componentsize manually, but that would be stupidly long. If I use the replace function and change all MaximumComponentSize to 25, that would change 17000 entries (most of them not being engines). That's what is bugging me right now.

Edit: But I seen some mods today that just patch things rather than edit the xml files, I guess that is something I should look into.
User avatar
frankycl
Posts: 401
Joined: Mon Mar 01, 2021 5:16 pm

Re: Some of the XML files are huge, Notepad++ don't cut it.

Post by frankycl »

Foraven wrote: Wed Aug 31, 2022 9:25 pm If I want to change all engine related MaximumComponentSize to 25, I can't use the replace function in Notepad ++. I can change them one by one, finding the <Type>Engine</Type> then changing each Componentsize manually, but that would be stupidly long. If I use the replace function and change all MaximumComponentSize to 25, that would change 17000 entries (most of them not being engines). That's what is bugging me right now.
Ok, now I've understood your problem - and here is how you can do what you want (in Notepad++):

Search-entry (e.g.; without the ""):

Code: Select all

"<Type>Engine</Type>\r\n        <MaximumComponentSize>120</MaximumComponentSize>"
Replace-entry (without the ""):

Code: Select all

"<Type>Engine</Type>\r\n        <MaximumComponentSize>20</MaximumComponentSize>"
(circled) checkbox: "extended"/"enhanced" (or something similar, above "regular expressions"; - I only have the German version and don't know what it is called in English)

-> "replace all" ... ;)

(please note: the number of empty spaces (between "\n" and <Maximum..." in the search- and replace-input-field must equal the number of empty spaces at the beginning of the second line !)
Foraven
Posts: 319
Joined: Mon Apr 20, 2009 1:32 am

Re: Some of the XML files are huge, Notepad++ don't cut it.

Post by Foraven »

frankycl wrote: Thu Sep 01, 2022 7:22 pm Ok, now I've understood your problem - and here is how you can do what you want (in Notepad++):

Search-entry (e.g.; without the ""):

Code: Select all

"<Type>Engine</Type>\r\n        <MaximumComponentSize>120</MaximumComponentSize>"
Replace-entry (without the ""):

Code: Select all

"<Type>Engine</Type>\r\n        <MaximumComponentSize>20</MaximumComponentSize>"
(circled) checkbox: "extended"/"enhanced" (or something similar, above "regular expressions"; - I only have the German version and don't know what it is called in English)

-> "replace all" ... ;)

(please note: the number of empty spaces (between "\n" and <Maximum..." in the search- and replace-input-field must equal the number of empty spaces at the beginning of the second line !)
Thanks, I didn't know I could use "\n" in find or replace fields. I will try that.
User avatar
frankycl
Posts: 401
Joined: Mon Mar 01, 2021 5:16 pm

Re: Some of the XML files are huge, Notepad++ don't cut it.

Post by frankycl »

Foraven wrote: Fri Sep 02, 2022 11:11 am Thanks, I didn't know I could use "\n" in find or replace fields. I will try that.
Yes, do that - but don't forget: you can only use "\r\n" (you need both !) when you put "extended"/"enhanced" for search-mode as enabled additionally ! ;)
arvcran2
Posts: 2758
Joined: Fri Dec 11, 2020 3:17 pm

Re: Some of the XML files are huge, Notepad++ don't cut it.

Post by arvcran2 »

Conceptually I think a relational data base with SQL support may be better method than using a text editor.

Of course, there are some hurdles to jump in order to get there! Defining a DDL from the XML description files, creating a database, loading the XML data into the database, and publishing the XML files from the data base after being modified.
User avatar
frankycl
Posts: 401
Joined: Mon Mar 01, 2021 5:16 pm

Re: Some of the XML files are huge, Notepad++ don't cut it.

Post by frankycl »

arvcran2 wrote: Tue Oct 11, 2022 2:59 pm Conceptually I think a relational data base with SQL support may be better method than using a text editor.

Of course, there are some hurdles to jump in order to get there! Defining a DDL from the XML description files, creating a database, loading the XML data into the database, and publishing the XML files from the data base after being modified.
Hm, sounds interesting... can you do that ?
(I guess it's similar to what Eventure does with his Mod-Maker tool ;-))
User avatar
rxnnxs
Posts: 503
Joined: Sat Jun 01, 2013 10:25 am
Location: what goes on
Contact:

Re: Some of the XML files are huge, Notepad++ don't cut it.

Post by rxnnxs »

How are you coming along? What do you use, what have you modded so far?
Post Reply

Return to “Design and Modding”