So throughout my time spent scripting things for CMANO/CMO I've often had to implement costly (performance wise) or inefficient workarounds for certain cases when it comes to state preservation across saves when dealing more than just trivial things.
Take an example where I have a table of data, or a table of my own objects, that I need to persist across saves. When it comes to anything more than the most simplest of settings the solution for those that do not have access to the io namespace or external modules (non-professional users) is to "stringify" the table(s) and save the result(s) to key(s) with ScenEdit_SetKeyValue. Then Upon scene-load have my script look for and load it back with ScenEdit_GetKeyValue and de-stringify them back to tables. This itself all works fine enough, even with multiple large tables in my experience, and thanks to lz4 it doesn't even bloat the file too bad most of the time as there is usually lots of duplicated patterns\entries in ones tables.
Now the but
A prime and re-occurring example of where a challenge arises though is not knowing when a user or the system has triggered a save event. At present since I never know when someone has triggered a 'save', be it an autosave or a manual one. The only workaround for making sure things are in sync that I've come up with is either re-saving to the storage key either every second, on every change, or every xx seconds and live with the fact my stored data could actually be xx amount of time out of date. The context again here isn't a setting or two or small table with a couple properties, it's far bigger stuff all requiring strigify and burning a dozen (or more) ms.
Saving every second is really wasteful of cpu time every second and in some cases, depending on what else you might be doing that is perf-heavy in that game cycle\second so can be just impractical at times in a med-large scene. Re-saving on-changes works pretty well in certain cases, again so long as the changes aren't constant or large or both. Saving state every XX seconds blanketly and "just dealing with out of date data on re-load" when dealing with a large amount of data is often a middle ground I'm choosing to avoid wasting cycles per-second slowing down the game. But often comes with additional code and complexity to re-validate all the stored data on scene load, or as part of function calls before using said data. Now here and there some of that is needed anyway, but often it is otherwise unnecessary bloat added only to account for possible timelapse in actual state vs key-saved state. The other hybrid is every second or x-seconds but _only_ if a dirty flag has been set by some other event (unit destruction, unit damaged,etc or mycode), but it doesn't always work well for everything and adds otherwise unnecessary events\triggers\actions, and dirty flag setting throughout the code.
A simple solution and way to avoid all of this I was thinking would be if there was a callback of sorts to LUA that was available to us for when a save event is triggered.
Such that at the beginning of any save process (be it manual save,manual save-as, or autosave), before the xml is generated, a properly try\catch surrounded synchronous call is added to execute a LUA chunk that simply does a "CMANO.SaveEvent();" or "CMANO.SaveEvent(iSaveTypeCodeHere);" or whatever reserved namespace entry the devs choose. Upon an exception it simply logs the error to the log (preferable the lua log) and continues on unabated with the save, and upon success return it continues on with the normal save process. I think a similar "PreLoadEvent()" or "UnloadEvent()" called prior to xml scene being loaded, and especially before the existing on scene-load event also has some value though it's less obvious why and less a concern, but I'll explain that later. Now obviously these could both be just implemented as first-class trigger types, but that may also be a more involved a process so I offer up the above as simpler option.
Doing such would provide authors the ability to tap into this by simply redirecting that function to their own and avoid all the unnecessary stuff that comes with the aforementioned workarounds and the designing around what is missing. Example of how I would use it:
print("save triggered: typecode " .. tostring(iSaveType));
CMANO.SaveEvent = gKH.SaveEvent;
Now I can do all sort of things at all sorts of timings without having to worry about them being in-sync upon a re-load of the scene, nor am I forced to take the stringify perf penalty on medium to large tables or object collections unless it's actually needed because of a save event.
I'm wondering what others think about the need for this or maybe have thoughts on better implementations, or if there is some gotcha here I am missing. I'm surprised this hasn't been raised before in fact I'm annoyed with myself for not raising it prior to now. I'm also curious if others have come up with other workarounds.
The aforementioned is just a "down and dirty" way to add some ability with what i think are minimal changes using what already exists, with what I think would be minimal risk, and minimal needed maintenance. It could of course as already mentioned be implemented as part of the existing event system by adding an "OnSceneSave" trigger type, though that's probably more work touching a wider part of the code-base (events\trigger create\delete\update etc), and more places for something to go wrong.
I also recognize that maybe I'm one of only a handful of people that would really take advantage of this enhancement, but I see it as a enabler for more advanced scripts for both commercial and professional customers. Also it sort of lowers the barrier to entry on the scripting front in general for more involved projects such that one doesn't have to be as concerned or as knowledgeable about managing the persisted state vs performance, particularly so if implemented as a full trigger-type. Conceptually if this matured to a trigger type it imho offers a more intuitive path for setting keys as part of a 'save' event than remembering to do it elsewhere throughout your scripts whenever some global variable or table is updated.
Similar CMANO.PreLoadEvent() (again name it whatever) value explained.
Readers are going to say why would that be needed? You can just clear your state in an on-scene-load lua script right? Well yes, you certainly can and should, as it relates to your own stuff that you know about.
But now consider the case where you have your own large library, and throw in some active LUA co-routines (yes they do work in CMO quite well!). You load up my scene (we'll call it Scene-A) you play all, my functions and var are created (hopefully under their own namespace to help avoid any conflicts), co-routines are created and work in the semi-background doing there things on the scene at intervals, all is well and good. You now want to play another scene (without restarting the game\exe), you load up Scene-B, you play scene-B. Well Scene-A's functions and co-routines (while not doing anything cause they're yielded) are still sitting in the background along with all the rest of my code and state from Scene A.
So unless Scene-B's author specifically knows to delete my crap then it's all still there because the environment is reused. Now most of the time this is largely a non-issue as the amount of data and funcs hanging around between scene loads isn't massive, and doesn't cause problems unless people are using the same named global vars and are not being wise about init'ing their own values on scene-load. But in could be, users like myself may have the same instance of CMO open for days at a time. Now is it really a problem that my co-routines and other functions are hanging piling up over time taking up what's usually only a few megabytes of memory? *shug* Probably not 99% of the time, but it would be nice if I could just hook in with a pre-load event and use it to properly force returns from co-routines, and nil most my functions and then trigger a garbage collection to leave the next guy as clean an environment as possible. Equally as mentioned above it could just be implemented as a 'OnSceneUnload' trigger, such that upon the process of loading a new file the old one's trigger fires just before actually loading the new one.
print(preload event triggered: memory used before " .. collectgarbage("count"))
print("preload event: force returning from co-routines and other cleanup tasks");
gKH = nil;
print(preload event: memory used after GC cycle: " .. collectgarbage("count"))
CMANO.PreLoadEvent = gKH.PreLoadEvent;
I fully admit I may very well be overthinking\overdoing it when it comes to a PreLoadEvent\UnloadEvent.
The SaveEvent is a much more pressing need on my wish list and the idea I'm most interested on feedback about. Also thoughts about it being implemented as full fledged triggers vs implemented outside the standard trigger system. The latter perhaps being more a "experimental feature" for now, maybe with full and proper trigger additions to follow later as developer time permits? I could also just be uninformed and am over-estimating the lift involved with adding trigger types so maybe it's a moot part of the discussion\idea.
< Message edited by KnightHawk75 -- 11/5/2020 1:22:54 PM >