# Generate differs in 10 and 100 runs - [x] New "new" format - [x] Init must check if both files are the same; if so, new format; if not, new new format. - [x] `Validation` needs pairs of files (Generated/Reference); assume `short` is always present, long is optional - [x] "update" can read the Validation directly - [x] If the generated file doesn't exist in the long run, ask again - [x] In "init", update the reference (this should be done inside `Generator`)\ # 0.20 Documentation - Rundir - Reduction - Small-Fail/Large-Fail - What is the absolute and relative differences # Not reducing - ===THIS MAY BE UNNECESSARY WITH THE NEW CONFIG FORMAT=== - `caseSetupDict.static`, maybe? - When there is no reduction, there is no need to build the short/long run things - The old format supports that, we just need to figure out when there are no reductions in `init` to generate the old format. - There is also a suggestion to change the run-dir when there is no reduction, which I'm not sure if possible due the other of "Expander"/"Reducer"... - Also, would need to have, in meta information, to have both (one option) rundirs. - This would help the other point of keeping the run-dir static for init, update and run. - Expanding all run-dirs at the start would save a few runs too. - In the pipeline, the sequence is Expander, Reducer, Loader. - The Expander doesn't know if the case can be reduced or not, and can't make a judgement if we need 1 or 2 run-dirs. - Maybe Reducer could have a "reducer probability" kinda of thing that Expander could call to find out the number of run-dirs. - Function returns: caseSetupDicts that can be reduced but are not continuations, caseSetupDicts that are continuations and caseSetupDicts that can't be reduced. I'd like to make it an iterator, but I guess returning a Vec is good enough (specially since we need to sort the files in the file system to find out the continuations). - Problem: Variations need to be sorta-kinda expanded, 'cause they have more/replace caseSetupDicts - So Reducer needs to know about variations and the merge space from the Expander, and Expander needs to know about reductability from the Reducer. - (All this to know how many run-dirs we need...) - Default doesn't have long run (all caseSetupDicts are static); variation have short and long run (at least oneCaseSetupDict is reducible). - Rename the run-dir after expansion, if found anything that can be reduced? - Expand the short and long run-dirs always, and then just ignore/delete the long one if the case isn't reducible? - We need to remove it if it doesn't have a long validation either. - Should we follow the same idea for controlDicts? - **Need to document this properly** [[#0.20 Documentation]] # system/validationDict Idea for a Foam-formated file for Verify ```c++ // Execution is run in sequence: If the first run ("quick") fails, then the // second run ("short") is run, and if that fails, the third is run ("long"), // and so on. On the other hand, the execution of the runs stops on the first // full success. // // This example is a bit of a stretch -- why would be want to run the same case // 4 different times? -- but it shows that, if that necessity appears to run // any different number of runs in the future, we can support it. It also shows // that one can have mixed reduced and non-reduce runs (e.g., there could be // just one run without reduction, or just one reduced run, and so on.) // We could also add support in Verify to only execute runs with a specific // name, allowing to run all examples in their "infinite" (non-reduced) mode // (as long as they all name their runs with no steps "infinite", that is). // // Names are free form, and we could use them to defined the run-dir, e.g., // "short" validation will be run in ".run-short", "long" in ".run-long" and // so on. runs ( // This is a quick test: The example is reduced to just 2 timesteps, and // we check the resulting file against its MD5 hash. quick { steps 2; // Any other key name that it is not reserved ("steps", "continuations") // represents a file to be compared. checks ( { result "postProcessing/2/qux"; md5 "123123"; } ) } // In case the MD5 fails, the case is run again, but this time reduced to // 10 timesteps. This run also includes continuations, by changing the // listed files to make one follow the timesteps of the previous. The run // will fail if relative or absolute differences are above 0. short { // *********************************************** // FUTURE IDEAS // *********************************************** // Workspace reuse: If "restart" and there is a rundir, it will be // deleted and a new one build; if "reuse" and there is a rundir, // nothing will be changed (if it doesn't, it will be created anyway). workspace reuse; // restart (default), reuse // Changes to be done before running the case. This is not applied in // case the workspace is being reused (in other words, this is done // only on "restart"). changes { "foamFile" { "system/controlDict/timestep" 0.2; } } // Commands that are run to prepare to run the example commands ( "decomposeParDict" "helyxSolve -force" ) // Post executions: This is actually the old "comparer", but now using // something different post ( ( comparer tsv; files "postProcessing/10/foo" "reference/10/foo"; variables "phi"; ) ( comparer md5; file "postProcessing/" ) ) // Execution: defines executions. Requires another run name and its // expected result. "quick fails" means "execute this run if 'quick' // fails". It could be "execute quick pass", in which this step will // only be run if the "quick" step passes. // Without this entry, the run becomes a top-level and it is always // run. execute quick fails; // execute // *********************************************** // END OF FUTURE IDEAS // *********************************************** steps 10; // This makes the continuations explicit, by specificing files that // form a single run. continuations ( "system/caseSetupDict.initial" "system/caseSetupDict.continuation1" "system/caseSetupDict.continuation2" ); checks ( { result "postProcessing/10/foo"; reference "verification/10/foo"; variables "phi" "Tmean"; // No tolerances mean "it will fail if the values are // not the same as they are in the reference". } { // A different file, with different variables. result "postProcessing/10/bar"; reference "verification/10/bar"; variables "Tmin" "Tmax"; } ) } // If in 10 timestemps the values do not match the reference file, a 100 // timesteps run is done. There is no continuation, and for this run to // fail, both absolute and relative tolerances must be above the designed // threshold. long { steps 100; // This file only appears in the 100 runs, so we can't have a global // list of files not attached to steps (although it is the same "foo" // from 10 steps). checks ( { result "postProcessing/100/foo"; reference "verification/100/foo"; variables "phi" "Tmean"; } { result "postProcessing/100/bar"; reference "verification/100/bar"; variables "Tmin" "Tmax"; } ) restart { steps 2; checks ( ... ) } } // If the 100 timesteps fails, then we run the example a 4th time, this // time without any reductions (the "steps" property is not set in this // run). There are no continations either. "run till completion" { checks ( { result "postProcessing/20000/baz"; reference "verification/20000/baz"; variables "integral"; absolute 20; relative 20; // no operator means OR, so example will fail if the // absolute different is above 20 OR the relative // difference is above 20. } ) } ); // This is used for filtering. tags "GIB" "AES" "compressible"; // Only present if the example can't be run in some platform. unstable true; operatingSystem windows; // valid values: "windows", "linux", "all" reason "Allrun uses Python, and Python isn't usually available on Windows"; ``` Pest: ```pest foam = { SOI ~ entry ~ EOI } // general stuff quote = _{ "\"" } semicolon = _{ ";" } braces_open = _{ "{" } braces_close = _ { "}" } parentheses_open = _{ "(" } parentheses_close = _{ ")" } specials = { semicolon | braces_open | braces_close | parentheses_open | parentheses_close | WHITESPACE } WHITESPACE = _{ " " | "\t" | NEWLINE } COMMENT = _{ "//" ~ (!NEWLINE ~ ANY)+ } // somewhat complex structures single_word = @{ (!specials ~ ANY)+ } quoted_word = @{ quote ~ (!quote ~ ANY)* ~ quote } keyword = { quoted_word | single_word } value = { keyword } // main foam stuff entry = { (dictionary | list | attribution)* } attribution = { keyword ~ value+ ~ semicolon } dictionary = { keyword ~ braces_open ~ entry ~ braces_close } list = { keyword ~ parentheses_open ~ value+ ~ parentheses_close ~ semicolon } ``` Parsed: ``` - foam - entry - list - keyword > single_word: "variables" - value > keyword > quoted_word: "\"phi\"" - value > keyword > quoted_word: "\"meanT\"" - list - keyword > single_word: "runs" - dictionary - keyword > single_word: "quick" - entry - attribution - keyword > single_word: "steps" - value > keyword > single_word: "2" - attribution - keyword > single_word: "generatedFile" - value > keyword > quoted_word: "\"postProcessing/2/blah\"" - dictionary - keyword > single_word: "failIf" - entry > attribution - keyword > single_word: "md5Differs" - value > keyword > quoted_word: "\"123123\"" - dictionary - keyword > single_word: "short" - entry - attribution - keyword > single_word: "steps" - value > keyword > single_word: "10" - attribution - keyword > single_word: "generatedFile" - value > keyword > quoted_word: "\"postProcessing/10/blah\"" - attribution - keyword > single_word: "referenceFile" - value > keyword > quoted_word: "\"verification/10/blah\"" - list - keyword > single_word: "continuations" - value > keyword > quoted_word: "\"system/caseSetupDict.initial\"" - value > keyword > quoted_word: "\"system/caseSetupDict.continuation1\"" - value > keyword > quoted_word: "\"system/caseSetupDict.continuation2\"" - dictionary - keyword > single_word: "long" - entry - attribution - keyword > single_word: "steps" - value > keyword > single_word: "100" - attribution - keyword > single_word: "generatedFile" - value > keyword > quoted_word: "\"postProcessing/100/blah\"" - attribution - keyword > single_word: "referenceFile" - value > keyword > quoted_word: "\"verification/100/blah\"" - dictionary - keyword > single_word: "failIf" - entry - attribution - keyword > single_word: "absolute" - value > keyword > single_word: "10" - attribution - keyword > single_word: "relative" - value > keyword > single_word: "10" - attribution - keyword > single_word: "operator" - value > keyword > single_word: "and" - dictionary - keyword > single_word: "infinite" - entry - attribution - keyword > single_word: "generatedFile" - value > keyword > quoted_word: "\"postProcessing/20000/blah;\"" - value > keyword > single_word: "referenceFile" - value > keyword > quoted_word: "\"verification/20000/blah\"" - dictionary - keyword > single_word: "failIf" - entry - attribution - keyword > single_word: "absolute" - value > keyword > single_word: "20" - attribution - keyword > single_word: "relative" - value > keyword > single_word: "20" - list - keyword > single_word: "tags" - value > keyword > quoted_word: "\"GIB\"" - value > keyword > quoted_word: "\"AES\"" - value > keyword > quoted_word: "\"compressible\"" - dictionary - keyword > single_word: "unstable" - entry - attribution - keyword > single_word: "operatingSystem" - value > keyword > single_word: "windows" - attribution - keyword > single_word: "reason" - value > keyword > quoted_word: "\"Allrun uses Python, and Python isn't usually available on Windows\"" - EOI: "" ``` Migration plans? - Side pipeline? - Current pipeline still exists, a new pipeline is build next to it with the new actors, with the new structures and "finder" sends requests depending on found files. - What about reporter? - No actors? - New design is in early stages and I'm not sure how long till they are both feature compatible. - Also, how about the current tests? Intermediate format? - We have nothing - We find a dictionary - That's a run, so we need to keep this in a list - In the run, we need to capture each content - It is "steps"? Int - It is 'failIf'? process dic - Now how it is "run"? - How do we fill the blanks while loading? # Logos ```rust use logos::Logos; #[allow(dead_code)] #[derive(Logos, Debug)] #[logos(skip r"[ \t\n\r]")] pub(crate) enum Token<'a> { #[regex(r#"\/\*[^\/\*]*\*\/"#, |lex| lex.slice())] MultilineComment(&'a str), #[regex(r#""[^"]+""#, |lex| lex.slice().trim_start_matches('"').trim_end_matches('"'))] #[regex("[a-zA-Z0-9]+", |lex| lex.slice())] Keyword(&'a str), #[regex(r#"//[^\n]*"#, |lex| lex.slice())] Comment(&'a str), #[token(";")] End, #[token("{")] DictStart, #[token("}")] DictEnd, #[token("(")] ListStart, #[token(")")] ListEnd, } ``` ``` variable "value with quotes"; ``` Keyword("variable") => Keyword("value with quotes) => End ``` dict { dict { var value; }} ``` Keyword("dict") => DictStart => Keyword("dict) => DictStart => Keyword("var") => Keyword("value) => End => DictEnd => DictEnd So: - Keyword (always) - if next is: - DictStart: Start dict - ListStart: Start List - Keyword: Grab keyword, make valuelist loop { - Grab keyword (key) - Grab next element: - Comment? None - MultilineComment: None - ListStart: process_list (consume token, anyway) - ListEnd: ERROR - DictStart: process_dictionary (consume token, anyway) - DictEnd: Empty dictionary, ERROR - Keyword: assume values, start Vec, add keyword, consume till "End" - End: ERROR - Grab next element: - None: Complete - DictEnd: This dictionary is complete, return it - Anything else: } # Post lexer work - [ ] grep filter on expansion - [ ] Runner with comparer - Spawn tasks? How to join? - Also, since the executor follows one run, and the comparer follow that run, does it matter using tasks? - [x] Build "rundir" when loading Verification - [ ] Variations?!?! - Expand the whole case at the start. - Load verification from expanded dirs. - This is what the "Loader" was doing before, which I inadvertently merged with the verification loader. - Order is mixed: The whole example can be expanded into their runs, but the execution is per non-expanded content (default, variations) - Need to send, into the pipeline, the default, variations and the replicated Verification for each. - Problem: The variation may contain different steps, anyway. So need to "expand" without having the verification info beforehand. - What if: - Expand default run into `.run-default` - Expand variations into `.run-[variationname]` - Inside each of those, expand the case (default won't expand, but still....) - We could say it is not possible to add a new verificationDict into variations, but I do like the idea of having a different content for it. - Ok, ignoring this for now 'cause it fucks up my thinking. - Expand by run on default, send variations down the pipeline\ # Named dicts ``` list ( name1 { inner 1; } name2 { inner 2; } ) ``` ``` Dictionary({ "list": [ List( [Value("name1"), Dictionary({"inner": [Value("1")]}), Value("name2"), Dictionary({"inner": [Value("2")]}) ] ) ] }) ``` # "Canonical" Paths We have an issue with the "run again", as if you run in a directory, it will show, in the report, the example name again -- which is not what we need. If "helyxVerify run ." is in an example, the report should show "." again. If "helyxVerify run" is not an example, the report should show the paths of the failed cases. Report needs to get the example path, and "run again" should use a HashSet. # Auto-Continuation ```c++ runs ( long { steps 100; checks ( { result "postProcessing/100/foo"; reference "verification/100/foo"; variables "phi" "Tmean"; } ); continuations ( "system/caseSetupDict" "system/caseSetupDict2" ); // New option: postSuccess { extraSteps 2; checks ( { result "postProcessing/102/foo"; reference "verification/long/continuation/100/foo"; variables "phi" "Tmean"; } ) } } ) ```