Review notes¶
What we noticed in the model while building the other pages, with a suggested change for each item. The model's authors describe it as a scaffold, incomplete by design, and invite others to critique and extend it (see About); these notes are offered in that spirit, in the hope that some of them are useful upstream.
The items are of different kinds. Some follow from the language: what SysML v2 makes a declaration mean, or what evaluating it gives. Others measure the model against goals it does not claim, such as being fully executable or completely traced, and are suggestions rather than errors. The register says which is which, and the checks are re-run every time the pages are built. Our checks can be wrong too; corrections are welcome as Sysight issues.
Completeness scorecard¶
Per layer, how much of the model is documented, identified and traced, computed from the model itself.
Use it to see where coverage could grow, keeping in mind that the model is a scaffold by design.
| Layer | Check | Covered | Share | ||
|---|---|---|---|---|---|
| Purpose | Definitions with documentation | 37 / 73 | 51% | ▲ partial | |
| Capabilities refining a stakeholder need | 25 / 25 | 100% | ✓ complete | ||
| Stakeholder needs refined by a capability | 22 / 22 | 100% | ✓ complete | ||
| Operational | Definitions with documentation | 39 / 39 | 100% | ✓ complete | |
| Operations refined by a function | 39 / 39 | 100% | ✓ complete | ||
| Operations satisfying a mission requirement | 18 / 39 | 46% | ▲ partial | ||
| Functional | Definitions with documentation | 75 / 75 | 100% | ✓ complete | |
| Functions allocated to a logical component | 51 / 69 | 74% | ▲ partial | ||
| Functions refining an operation | 52 / 69 | 75% | ▲ partial | ||
| Leaf functions satisfying a functional requirement | 51 / 52 | 98% | ▲ partial | ||
| Logical | Definitions with documentation | 5 / 6 | 83% | ▲ partial | |
| Logical components specializing CoSMA LogicalComponent | 5 / 6 | 83% | ▲ partial | ||
| Technical | Definitions with documentation | 141 / 144 | 98% | ▲ partial | |
| Hardware designs satisfying a technical requirement (directly or through an individual) | 17 / 22 | 77% | ▲ partial | ||
| Requirements | Definitions with documentation | 294 / 294 | 100% | ✓ complete | |
| Functional requirements refining a higher level | 111 / 112 | 99% | ▲ partial | ||
| Functional requirements satisfied | 112 / 112 | 100% | ✓ complete | ||
| Mission requirements refining a higher level | 43 / 43 | 100% | ✓ complete | ||
| Mission requirements satisfied | 43 / 43 | 100% | ✓ complete | ||
| Requirements with an identifier | 294 / 294 | 100% | ✓ complete | ||
| Satisfy statements whose requirement can be checked from the model | 0 / 272 | 0% | ▲ partial | ||
| Technical requirements refining a higher level | 106 / 117 | 91% | ▲ partial | ||
| Technical requirements satisfied | 103 / 117 | 88% | ▲ partial | ||
| Analysis | Definitions with documentation | 15 / 17 | 88% | ▲ partial | |
| Analyses with a specific subject | 4 / 5 | 80% | ▲ partial | ||
| Analysis results the model can compute | 10 / 26 | 38% | ▲ partial | ||
| Asserted constraints that hold (excluding time relations) | 0 / 3 | 0% | ▲ partial | ||
| Execution | Definitions with documentation | 0 / 1 | 0% | ▲ partial | |
| CoSMA framework | Definitions with documentation | 3 / 33 | 9% | ▲ partial |
- Analysis: asserted constraints that hold (excluding time relations) — 0 of 3 (0%).
- Requirements: satisfy statements whose requirement can be checked from the model — 0 of 272 (0%).
- CoSMA framework: definitions with documentation — 3 of 33 (9%).
- Analysis: analysis results the model can compute — 10 of 26 (38%).
- Operational: operations satisfying a mission requirement — 18 of 39 (46%).
- Purpose: definitions with documentation — 37 of 73 (51%).
- Functional: functions allocated to a logical component — 51 of 69 (74%).
- Functional: functions refining an operation — 52 of 69 (75%).
- Technical: hardware designs satisfying a technical requirement (directly or through an individual) — 17 of 22 (77%).
Issue register¶
What the checks and our review found, each with its severity, its basis, the elements involved and a suggested change. Severity is high when an item changes what the model means or what its analyses compute, medium when it affects a trace, a convention or a check, low when it is presentation. The basis says what an item rests on: SysML v2 semantics and evaluation follow from the language and from evaluating the model as written; engineering review from the physics or the mission record; CoSMA convention, executable-model goal and good practice measure the model against a method or a goal, and are suggestions rather than errors.
Use it to choose changes to make or propose upstream; rows marked detected are re-checked against the model every time this page is built, and disappear once addressed. IDs stay with their item.
| ID | Severity | Category | Basis | Issue | Elements | Suggested change | Check | Page |
|---|---|---|---|---|---|---|---|---|
| Q-01 | high | structure | SysML v2 semantics | Subsetting written after a body declares a separate anonymous feature. Declarations such as requirement goToMoon : Goal { … } :> goals; end at the closing brace, so the trailing :> goals; declares a new, unnamed reference instead of making the requirement a member of goals. The elements meant to join capabilityToGoals, concerns, goals, missions are therefore not members of the features meant to collect them. | 40 × :> concerns; 25 × :> capabilityToGoals; 17 × :> missions; 5 × :> goals | Move the subsetting into the declaration: requirement goToMoon : Goal :> goals { … }. | detected | Mission |
| Q-03 | high | analysis | SysML v2 semantics | An assertion that constrains nothing. assert constraint { overallMissionSuccess = true; } in Apollo11MissionSuccessAnalysis: with the semicolon, the body declares a new feature named overallMissionSuccess bound to true instead of stating a Boolean condition, so the assertion always holds. | Apollo11MissionSuccessAnalysis::3 | Write the condition as an expression: assert constraint { overallMissionSuccess } (or == true, without the semicolon). | detected | Budgets |
| Q-04 | high | analysis | SysML v2 semantics | Stage stacks in the delta-v analysis include the SI unit lumen. The delta-v analysis lists the lunar module as lm in 4 stacks (fullStack, secondStageStack, spacecraftStack, tliStack), but defines no feature named lm. Through import SI::* the name resolves to SI::lumen (short name lm), so the lunar module's mass is missing from every stack mass. | fullStack; secondStageStack; spacecraftStack; tliStack | Declare the lunar module in the analysis (lm = spacecraft.lm;) or use an unambiguous name. | detected | Budgets |
| Q-05 | high | analysis | SysML v2 semantics | calculateDeltaV is called without its standard-gravity argument. calculateDeltaV(isp, initialMass, finalMass) passes three of its 4 inputs (isp, g0, m0, mf). By position the initial mass binds to g0 and the final mass to m0; mf stays unbound. | ascentMargin_lunar; dv_csm; dv_lm_ascent; dv_lm_descent; landingMargin; loiMargin | Pass g0 explicitly, or call with named arguments (calculateDeltaV(isp = …, m0 = …, mf = …)) and give g0 a default. | detected | Budgets |
| Q-06 | high | analysis | evaluation | Mass roll-up has no parts to sum. CoSMA computes totalMass = mass + sum(subcomponents.totalMass), but no part usage subsets subcomponents, so every component's total mass equals its own mass and the roll-up does nothing. | CoSMAPackage subcomponents (no subsetting part) | Declare component parts as subcomponents (part sic : 'S-IC' :> subcomponents;). | detected | Budgets |
| Q-13 | high | analysis | SysML v2 semantics | The reliability analysis uses the Euler number of fluid dynamics, not e. calculateReliability computes eulerNumber^(-(failureRate * missionTime)). The name resolves to ISQCharacteristicNumbers::eulerNumber, a dimensionless characteristic number of fluid dynamics with no value, not the constant e = 2.718…, so no reliability can be computed. | cmReliability; smReliability | Use exp(-(failureRate * missionTime)) (or a constant for e defined in the model). | detected | Budgets |
| Q-14 | high | analysis | evaluation | An asserted constraint is false. Evaluated as written, the constraint an analysis asserts does not hold, so the analysis reports a failure. | SystemPowerAnalysis: powerMargin > 0[W] with powerMargin = 0 W | Check the inputs the assertion depends on (for the power analysis: bind its subject to the mission system). | detected | Budgets |
| Q-02 | medium | structure | SysML v2 semantics | Stakeholder attributes set by subsetting instead of redefinition. Each of the 12 stakeholders writes :> influenceLevel = …; and :> influenceKind = …;. Subsetting adds a new feature with that value; the inherited influenceLevel and influenceKind themselves stay unset. | 12 × :> influenceKind = …; 12 × :> influenceLevel = … | Redefine the inherited features: :>> influenceLevel = StakeholderInfluenceLevel::high;. | detected | Mission |
| Q-27 | medium | structure | executable-model goal | Phase exit criteria are prose, not guards. The state machine's transitions state their exit criteria in documentation (doc /* if "…" and … */), so they cannot be checked or simulated. | 13 transitions of apollo11Phases | Express the criteria as guard expressions over attributes (if launchEscapeJettisoned and …). | detected | Operations |
| Q-28 | medium | structure | executable-model goal | Phase-completion notifications are never sent. 13 of the 14 phase transitions wait for a …PhaseCompletedNotification, but no action sends one, so the state machine cannot advance in a simulation. | 13 transitions wait for a notification; 0 send actions | End each phase's operation sequence with a send of its completion notification. | detected | Operations |
| Q-29 | medium | structure | CoSMA convention | Logical component not derived from the CoSMA base type. This part definition in the logical layer does not specialize CoSMA's LogicalComponent, so queries and rules over logical components miss it. | DataTransmissionNetwork | Add :> LogicalComponent to its declaration. | detected | Functions |
| Q-16 | medium | verification | executable-model goal | Requirements cannot yet be checked from the model. Evaluating each requirement for the element that satisfies it decides nothing: quantitative requirements compare an actual… attribute that is never bound (no design value or measurement is connected to it), and the other requirements state no constraint. The satisfy relationships therefore record allocation, which is ordinary SysML v2 traceability; checking them from the model is a further step. | 45 satisfy statements: the requirement's actual value is never bound; 227 satisfy statements: the requirement has no require constraint | Bind each quantitative requirement's actual value to the satisfying element's attribute (e.g. attribute :>> actualLiftoffThrust = stage1.thrust; in the satisfying context), and add require constraints to the requirements that should be checkable. | detected | Requirements |
| Q-20 | medium | traceability | CoSMA convention | Technical requirements outside the trace. These technical requirements neither refine a functional requirement nor appear in the system specification: nothing connects them to the rest of the model. | CLR-R055; CLR-R059; CLR-R061; CLR-R062; CLR-R063; CLR-R064; CLR-R065; CLR-R066; CLR-R070; CLR-R072; CLR-R074 | Add the refinement to the functional requirement they derive from and a requirement usage in SystemSpecificationPackage, or delete them. | detected | Requirements |
| Q-21 | medium | traceability | CoSMA convention | Specified requirements that nothing satisfies. These requirements are part of a specification but no element satisfies them. | CLR-R077; CLR-R104; CLR-R105 | Add the satisfy statements (or record why they are open). | detected | Requirements |
| Q-22 | medium | traceability | CoSMA convention | Functional requirements without a parent. These functional requirements refine nothing, so they do not trace to any mission requirement or capability. | FLR-R091 | Add the #refinement dependency to the mission requirement they come from. | detected | Requirements |
| Q-23 | medium | traceability | CoSMA convention | Functional requirements that skip the mission level. These functional requirements refine a capability directly instead of a mission requirement, which breaks the level-by-level chain. | FLR-R092 → AdaptabilityAndProblemSolving; FLR-R092 → HumanPerformanceUnderExtremeStress; FLR-R093 → ComplexLogisticsAndIntegration; FLR-R094 → CrewTrainingAndSimulation | Refine the corresponding mission requirement (add one if it is missing). | detected | Requirements |
| Q-24 | medium | traceability | CoSMA convention | Leaf functions with no functional requirement. No functional requirement is satisfied by these leaf functions. | ProvideIndividualLifesupport | Add the functional requirement, or merge the function into one that has requirements. | detected | Functions |
| Q-07 | medium | analysis | CoSMA convention | Instrument unit declares its own dryMass instead of a mass. SaturnVInstrumentUnit is not a fuelled component but declares a new attribute dryMass (1,950 kg) rather than redefining the inherited mass, so mass budgets based on mass miss it. | dryMass | Use attribute :>> mass = 1950 [kg];. | detected | Budgets |
| Q-08 | medium | analysis | engineering review | The ascent check leaves out the S-IVB's orbit-insertion burn. The ascent-to-orbit check adds only the first two stages' delta-v and compares it with the whole requirement to reach orbit; the S-IVB's first burn, which completes orbit insertion, is credited to translunar injection instead. The model cannot evaluate the margin today (ascentMargin needs instrumentUnit.mass, which has no value (none in SaturnVInstrumentUnit)); the Technical budgets page computes the corrected figures. | ascentCapability = dv_stage1 + dv_stage2 against dv_required_leo = 9400 m/s | Split the S-IVB propellant between orbit insertion and TLI, or compare the ascent requirement with all three stages. | detected | Budgets |
| Q-09 | medium | analysis | SysML v2 semantics | naturalLogarithm is declared with two inputs. calc <ln> naturalLogarithm { in x; in y; … } takes two inputs, but its call site passes one; y is never bound. | CoSMAQuantitiesAndUnitsPackage::naturalLogarithm (2 inputs) | Declare a single input, or use the standard library's NumericalFunctions / ScalarFunctions equivalent. | detected | Budgets |
| Q-10 | medium | analysis | engineering review | Gravitational parameter typed as a force. mu_Earth and mu_Moon subset ISQ::force, but a gravitational parameter is in m³/s², so unit checking cannot catch an error in these inputs. | mu_Earth; mu_Moon | Define a quantity for the gravitational parameter (m³⋅s⁻²) in CoSMAQuantitiesAndUnitsPackage and use it. | detected | Budgets |
| Q-11 | medium | analysis | evaluation | Power analysis has an unbound, generic subject. Apollo11MissionSystemPowerAnalysis declares a subject typed by a generic CoSMA type (subject apollo11MissionSystem : System;) with no value: any system, not the Apollo 11 mission system, so the analysis has nothing concrete to evaluate. | Apollo11MissionSystemPowerAnalysis | Bind the subject: subject apollo11MissionSystem : Apollo11MissionSystem = …;. | detected | Budgets |
| Q-15 | medium | analysis | evaluation | An asserted constraint cannot be evaluated. The evaluator cannot decide the assertion because a value it depends on is missing from the model. | Apollo11MissionDeltaVBudgetAnalysis: ascentMargin needs instrumentUnit.mass, which has no value (none in SaturnVInstrumentUnit) | Supply the missing value named in the elements column. | detected | Budgets |
| Q-30 | low | structure | good practice | Initial and final phases modeled as ordinary states. initial and final are states typed by Initial and MissionComplete, not the state machine's entry and done, so tools do not recognize where the machine starts and ends. | final; initial | Use entry; then preparation; and a transition to done. | detected | Operations |
| Q-31 | low | structure | good practice | Context definitions that nothing uses. These context definitions are defined but nothing specializes or uses them, so the mission context does not include them. | CislunarSpaceEnvironment; EarthEnvironment; LaunchSite; LunarEnvironment | Add them to Apollo11MissionContext, or move them to a library package. | detected | Mission |
| Q-19 | low | traceability | good practice | Identical satisfy statements. The same requirement is satisfied by the same element twice, which inflates satisfaction counts. | clr-R035 by plss; hlr-R065 by monitorSpacecraftSystems; clr-R050 by lunarModuleDescentStage; clr-R008 by instrumentUnit; clr-R024 by plss; hlr-R067 by monitorSpacecraftSystems; clr-R025 by plss | Delete the duplicate satisfy statements. | detected | Requirements |
| Q-25 | low | traceability | CoSMA convention | Requirements attached to only one use of a shared function. These functions are used in more than one function of the breakdown, but the function specification satisfies their requirements with only some of the usages. | MonitorDownlinkedTelemetry: in TraverseToMoon, not in TraverseToEarth; MonitorOnboardSystems: in TraverseToMoon, not in TraverseToEarth; TrackAndRelayGroundSignals: in TraverseToMoon, not in TraverseToEarth; TransmitAndReceiveSpacecraftSignals: in TraverseToMoon, not in TraverseToEarth | Satisfy the requirements with the function definitions, or with every usage. | detected | Functions |
| Q-26 | low | traceability | good practice | Vehicles record the missions they flew only in prose. Which mission a vehicle individual flew on is stated only in its documentation; only CSM-107, LM-5, SA-506 are used by a part of the model. | 45 vehicles, e.g. CM-099, CSM-101, CSM-103, CSM-104 | Model the assignment (e.g. the mission's system parts typed by the individuals). | detected | Physical |
| Q-17 | low | analysis | good practice | Analysis features that are computed but never used. Some features of the analyses are given values but nothing refers to them (e.g. les, tliStack). | les; tliStack | Remove them, or use them where they were intended (the TLI stack). | detected | Budgets |
| Q-18 | low | analysis | engineering review | Recorded splashdown time differs from the planned duration. The as-flown record puts splashdown at a different mission time than the mission's plannedDuration; one of the two values (or its source) needs checking. | plannedDuration 195.3 h; atSplashdown missionTime 191.8 h | Reconcile the values against the Apollo 11 mission report and cite the source. | detected | Timeline |
| Q-32 | low | documentation | good practice | Misspelled feature name. CoSMA's Mission declares requiredCapabilites (missing an *i*); every mission inherits the misspelling. | requiredCapabilites | Rename it to requiredCapabilities (the language server can rename all references). | detected | Navigator |
| Q-33 | low | documentation | good practice | Stray text inside documentation. The documentation of concern4 reads “Adeq :> concerns;uate Training & Preparation”, which looks like text left over from an edit. | concern4 | Correct the documentation text. | detected | Mission |
| Q-34 | low | documentation | good practice | Description written as a comment instead of documentation. The description is a plain /* */ comment, which tools do not treat as the element's documentation. | Spacecraft | Turn it into doc /* … */. | detected | Functions |
| Q-35 | low | documentation | good practice | Gaps in the requirement numbering. Requirement identifiers skip numbers; the numbers may have been retired, or requirements may be missing. | HLR: 26 unused numbers between HLR-R001 and HLR-R069 | Check whether requirements are missing; otherwise record that the numbers are retired. | detected | Requirements |
Pages with the context of each issue: Mission, Operations, Timeline, Functions, Physical, Budgets, Requirements, Navigator.
Suggested changes¶
The changes we think would help the model most, in the order we would make them, each covering one or more register entries.
Use it to decide what to take up; each item lists the register IDs it addresses.
- Let the analyses compute. Declare the lunar module in the delta-v stacks, pass
g0(or use named arguments), givenaturalLogarithmone input, type the gravitational parameters correctly, credit the S-IVB's orbit-insertion burn to the ascent, and use e (exp) in the reliability calculation. Today the model computes 10 of 26 of its analysis results. (Q-04, Q-05, Q-08, Q-09, Q-10, Q-13) - Align the subsettings with their intent. Move the trailing
:> …;into the declarations (87 anonymous features today), and redefine, rather than subset, the stakeholder attributes. The elements meant to joincapabilityToGoals,concerns,goals,missionsthen become members of them. (Q-01, Q-02) - Make the mass, power and success roll-ups compute. Subset
subcomponentswith the component parts, give the instrument unit amass, bind the power analysis to the mission system, and write the vacuous assertion as a condition. (Q-03, Q-06, Q-07, Q-11, Q-14, Q-15) - Close the requirement trace. Connect the 11 isolated technical requirements (or delete them), satisfy the 3 specified but unsatisfied requirements, remove the duplicate
satisfystatements, give the parentless functional requirements their mission requirement, route the refinements that skip the mission level through a mission requirement, satisfy shared functions' requirements with every usage, and give the leaf functions without requirements their functional requirements. (Q-19, Q-20, Q-21, Q-22, Q-23, Q-24, Q-25) - Make the concept of operations executable, if that is a goal. Write the phase exit criteria as guards, send the completion notifications, and use the state machine's entry and
done. (Q-27, Q-28, Q-30) - Housekeeping. Fix the misspelled
requiredCapabilites, correct the stray documentation text, turn the descriptive comment into documentation, decide on the unused context definitions, and model which mission each vehicle flew. (Q-26, Q-31, Q-32, Q-33, Q-34, Q-35)
What the model does not cover¶
The checks above judge the model against itself. A concept review also judges it against what a mission concept is expected to contain, and a review of this model by David Wagner found several topics missing. They are of two kinds: content the model could gain, because the historical record documents it, and content a model built after the fact cannot recover, because it records decisions as they were being made.
Topics a mission concept review expects and the historical record documents, with what the model holds for each (counted live) and the page where the content would appear.
Use it to plan what to add to the model; a row clears when the model gains the content.
| Topic | What a concept review expects | What the model holds | Where it would appear |
|---|---|---|---|
| Risk | A risk register with likelihood, consequence and mitigation, at the level of the stakeholder concerns. | No risk element; 'risk' appears only in the name of the definition MicrometeoroidPenetrationRisk. | Mission overview |
| Off-nominal operations | Abort modes and contingency paths, which explain designs such as the launch escape system. | 0 phases branch (the phase sequence is linear); abort appears in 4 requirements (AbortSequenceExecutionTime, CountdownAbortRequirement, CrewAscentAbortRequirement, LaunchAbortThresholds) and 3 other features (abortSignal, actualAbortExecutionTime, maxAbortExecutionTime), and in 0 phases, transitions or operations. | Concept of operations |
| Verification and validation | How each requirement will be verified (test, analysis, inspection), and the V&V campaign. | 0 verification cases. | Requirements and traceability |
| Schedule and work phasing | The project phases and milestones needed to deliver the system. | 0 project phases, milestones or schedules (the only schedules are the crew's: CrewActivitySchedulingRequirement, ManageCrewSchedules, manageCrewSchedules). | Programmatic resources |
| Estimate maturity | Mass and power as a current best estimate (CBE) and a maximum with growth allowance (MGA). | 0 growth or allowance attributes; each mass and power is a single value. | Technical budgets |
| Propellant bounds | The minimum propellant each stage needs, its tank capacity, and the margin between them. | 0 capacity or reserve attributes; 6 stages each state one propellantMass. | Technical budgets |
| Power and energy by phase | Peak power and total energy per phase, and the fuel-cell reactants that energy consumes. | 0 power values on a phase or an operation; power is one value per part: 8 loads and 5 sources. | Technical budgets |
| Ground systems | The ground systems developed for the program, in the system's breakdown. | 6 ground systems (CrawlerTransporter, LaunchControlCenter, LaunchSite, MissionControl, RecoveryForces, VehicleAssemblyBuilding), in ContextPackage; 0 in the technical layer. | Physical architecture |
| Physical geometry | Configuration drawings: shapes, dimensions and how the stack fits together. | 0 uses of the SysML shape or spatial libraries. | Physical architecture |
| Cost qualification | Costs in constant-year currency, phased by project phase, split between the program and the mission. | 4 lump-sum costs (manufacturingCost, operationsCost, personnelCost, researchAndDevelopmentCost) in $, with no base year or phasing. | Programmatic resources |
- Every value would need a source (NASA mission reports, press kits, budget histories), cited in the model: growth allowances, risk ratings or power profiles made up to fill a table would be worse than the gap.
- Kept in packages of their own, the additions would leave the original CoSMA layers unchanged and could be offered upstream as one contribution.
What a live project's model would hold that a retrospective one cannot, because it records decisions as they were being made.
Use it to read the other pages with these limits in mind, and avoid over-interpreting what is absent.
- Trades and discarded options (0 variations in the model): only the design that flew is modeled. Celebrated trades such as lunar-orbit rendezvous against direct ascent are documented, but the full set of goals and capabilities proposed and dropped along the way, and the cost and risk assessments that drove them, are not, so the capability map cannot show why each capability survived.
- Cost as it was obligated: a project tracks cost by phase as it is committed (R&D early, manufacturing in development, operations at the end). The model records its costs as lump sums for the mission, with no split by phase or year (its own comment: “All costs are estimations and more a best guess”).
- Requirements as written then: the model's requirements read as reconstructed for it rather than taken from Apollo's specifications, so their wording and completeness reflect the reconstruction; the gaps in their trace are gaps in the reconstruction.
- Several of the language findings (a subsetting written after a body,
=followed by;in a constraint, a short name such aslmresolving to a unit) are easy to write in the SysML v2 textual notation and hard to see by reading it. They say as much about what tools should flag as about the model. - To reproduce the findings, clone the model, install Sysight and run
sysight checkin the model's folder: it reports some of them (the asserted constraints, the unbound arguments, mismatched units) with their file and line. The other checks are SPARQL queries over the same model, whichsysight queryruns as well.