# Generate differs in 10 and 100 runs - [x] New "new" format - [x] Init must check if both files are the same; if so, new format; if not, new new format. - [x] `Validation` needs pairs of files (Generated/Reference); assume `short` is always present, long is optional - [x] "update" can read the Validation directly - [x] If the generated file doesn't exist in the long run, ask again - [x] In "init", update the reference (this should be done inside `Generator`)\ # 0.20 Documentation - Rundir - Reduction - Small-Fail/Large-Fail - What is the absolute and relative differences # Not reducing - ===THIS MAY BE UNNECESSARY WITH THE NEW CONFIG FORMAT=== - `caseSetupDict.static`, maybe? - When there is no reduction, there is no need to build the short/long run things - The old format supports that, we just need to figure out when there are no reductions in `init` to generate the old format. - There is also a suggestion to change the run-dir when there is no reduction, which I'm not sure if possible due the other of "Expander"/"Reducer"... - Also, would need to have, in meta information, to have both (one option) rundirs. - This would help the other point of keeping the run-dir static for init, update and run. - Expanding all run-dirs at the start would save a few runs too. - In the pipeline, the sequence is Expander, Reducer, Loader. - The Expander doesn't know if the case can be reduced or not, and can't make a judgement if we need 1 or 2 run-dirs. - Maybe Reducer could have a "reducer probability" kinda of thing that Expander could call to find out the number of run-dirs. - Function returns: caseSetupDicts that can be reduced but are not continuations, caseSetupDicts that are continuations and caseSetupDicts that can't be reduced. I'd like to make it an iterator, but I guess returning a Vec is good enough (specially since we need to sort the files in the file system to find out the continuations). - Problem: Variations need to be sorta-kinda expanded, 'cause they have more/replace caseSetupDicts - So Reducer needs to know about variations and the merge space from the Expander, and Expander needs to know about reductability from the Reducer. - (All this to know how many run-dirs we need...) - Default doesn't have long run (all caseSetupDicts are static); variation have short and long run (at least oneCaseSetupDict is reducible). - Rename the run-dir after expansion, if found anything that can be reduced? - Expand the short and long run-dirs always, and then just ignore/delete the long one if the case isn't reducible? - We need to remove it if it doesn't have a long validation either. - Should we follow the same idea for controlDicts? - **Need to document this properly** [[#0.20 Documentation]] # system/validationDict Idea for a Foam-formated file for Verify ```c++ // List of variables to be checked; optional, since MD5 validation does not // use any variables. variables ("phi" "meanT"); // Execution is run in sequence: If the first run ("quick") fails, then the // second run ("short") is run, and if that fails, the third is run ("long"), // and so on. On the other hand, the execution of the runs stops on the first // full success. // // This example is a bit of a stretch -- why would be want to run the same case // 4 different times? -- but it shows that, if that necessity appears to run // any different number of runs in the future, we can support it. It also shows // that one can have mixed reduced and non-reduce runs (e.g., there could be // just one run without reduction, or just one reduced run, and so on.) // We could also add support in Verify to only execute runs with a specific // name, allowing to run all examples in their "infinite" (non-reduced) mode // (as long as they all name their runs with no steps "infinite", that is). // // Names are free form, and we could use them to defined the run-dir, e.g., // "short" validation will be run in ".run-short", "long" in ".run-long" and // so on. runs ( // This is a quick test: The example is reduced to just 2 timesteps, and // we check the resulting file against its MD5 hash. quick { steps 2; generatedFile "postProcessing/2/blah"; // because the validation is in MD5 format, there is no need to have // a referenceFile failIf { md5Differs "123123"; } } // In case the MD5 fails, the case is run again, but this time reduced to // 10 timesteps. This run also includes continuations, by changing the // listed files to make one follow the timesteps of the previous. The run // will fail if relative or absolute differences are above 0. short { steps 10; generatedFile "postProcessing/10/blah"; referenceFile "verification/10/blah"; // No tolerances mean "it will warn if the values are not the same" // This makes the continuations explicit, by specificing files that // form a single run. continuations ( "system/caseSetupDict.initial" "system/caseSetupDict.continuation1" "system/caseSetupDict.continuation2" ); } // If in 10 timestemps the values do not match the reference file, a 100 // timesteps run is done. There is no continuation, and for this run to // fail, both absolute and relative tolerances must be above the designed // threshold. long { steps 100; generatedFile "postProcessing/100/blah"; referenceFile "verification/100/blah"; failIf { absolute 10; relative 10; // will fail is the absolute difference is above 10 AND the // relative difference is above 10; operator and; } } // If the 100 timesteps fails, then we run the example a 4th time, this // time without any reductions (the "steps" property is not set in this // run). There are no continations either. infinite { // With no steps, there is no reduction. // (maybe we need something to tell that there is no reduction // explicitly?) generatedFile "postProcessing/20000/blah;" referenceFile "verification/20000/blah"; failIf { absolute 20; relative 20; // no operator means OR, so example will fail if the absolute // different is above 20 OR the relative difference is above 20 } } ); // This is used for filtering. tags ("GIB" "AES" "compressible"); // Only present if the example can't be run in some platform. unstable { operatingSystem windows; // valid values: "windows", "linux", "all" reason "Allrun uses Python, and Python isn't usually available on Windows"; } ``` Pest: ```pest foam = { SOI ~ entry ~ EOI } // general stuff quote = _{ "\"" } semicolon = _{ ";" } braces_open = _{ "{" } braces_close = _ { "}" } parentheses_open = _{ "(" } parentheses_close = _{ ")" } specials = { semicolon | braces_open | braces_close | parentheses_open | parentheses_close | WHITESPACE } WHITESPACE = _{ " " | "\t" | NEWLINE } COMMENT = _{ "//" ~ (!NEWLINE ~ ANY)+ } // somewhat complex structures single_word = @{ (!specials ~ ANY)+ } quoted_word = @{ quote ~ (!quote ~ ANY)* ~ quote } keyword = { quoted_word | single_word } value = { keyword } // main foam stuff entry = { (dictionary | list | attribution)* } attribution = { keyword ~ value+ ~ semicolon } dictionary = { keyword ~ braces_open ~ entry ~ braces_close } list = { keyword ~ parentheses_open ~ value+ ~ parentheses_close ~ semicolon } ``` Parsed: ``` - foam - entry - list - keyword > single_word: "variables" - value > keyword > quoted_word: "\"phi\"" - value > keyword > quoted_word: "\"meanT\"" - list - keyword > single_word: "runs" - dictionary - keyword > single_word: "quick" - entry - attribution - keyword > single_word: "steps" - value > keyword > single_word: "2" - attribution - keyword > single_word: "generatedFile" - value > keyword > quoted_word: "\"postProcessing/2/blah\"" - dictionary - keyword > single_word: "failIf" - entry > attribution - keyword > single_word: "md5Differs" - value > keyword > quoted_word: "\"123123\"" - dictionary - keyword > single_word: "short" - entry - attribution - keyword > single_word: "steps" - value > keyword > single_word: "10" - attribution - keyword > single_word: "generatedFile" - value > keyword > quoted_word: "\"postProcessing/10/blah\"" - attribution - keyword > single_word: "referenceFile" - value > keyword > quoted_word: "\"verification/10/blah\"" - list - keyword > single_word: "continuations" - value > keyword > quoted_word: "\"system/caseSetupDict.initial\"" - value > keyword > quoted_word: "\"system/caseSetupDict.continuation1\"" - value > keyword > quoted_word: "\"system/caseSetupDict.continuation2\"" - dictionary - keyword > single_word: "long" - entry - attribution - keyword > single_word: "steps" - value > keyword > single_word: "100" - attribution - keyword > single_word: "generatedFile" - value > keyword > quoted_word: "\"postProcessing/100/blah\"" - attribution - keyword > single_word: "referenceFile" - value > keyword > quoted_word: "\"verification/100/blah\"" - dictionary - keyword > single_word: "failIf" - entry - attribution - keyword > single_word: "absolute" - value > keyword > single_word: "10" - attribution - keyword > single_word: "relative" - value > keyword > single_word: "10" - attribution - keyword > single_word: "operator" - value > keyword > single_word: "and" - dictionary - keyword > single_word: "infinite" - entry - attribution - keyword > single_word: "generatedFile" - value > keyword > quoted_word: "\"postProcessing/20000/blah;\"" - value > keyword > single_word: "referenceFile" - value > keyword > quoted_word: "\"verification/20000/blah\"" - dictionary - keyword > single_word: "failIf" - entry - attribution - keyword > single_word: "absolute" - value > keyword > single_word: "20" - attribution - keyword > single_word: "relative" - value > keyword > single_word: "20" - list - keyword > single_word: "tags" - value > keyword > quoted_word: "\"GIB\"" - value > keyword > quoted_word: "\"AES\"" - value > keyword > quoted_word: "\"compressible\"" - dictionary - keyword > single_word: "unstable" - entry - attribution - keyword > single_word: "operatingSystem" - value > keyword > single_word: "windows" - attribution - keyword > single_word: "reason" - value > keyword > quoted_word: "\"Allrun uses Python, and Python isn't usually available on Windows\"" - EOI: "" ``` Migration plans? - Side pipeline? - Current pipeline still exists, a new pipeline is build next to it with the new actors, with the new structures and "finder" sends requests depending on found files. - What about reporter? - No actors? - New design is in early stages and I'm not sure how long till they are both feature compatible. - Also, how about the current tests? Intermediate format? - We have nothing - We find a dictionary - That's a run, so we need to keep this in a list - In the run, we need to capture each content - It is "steps"? Int - It is 'failIf'? process dic - Now how it is "run"? - How do we fill the blanks while loading?