Data Reconciliation¶
Objective of Data Reconciliation¶
Data reconciliation aims at improving the estimation of the actual state of a physical system by combining: * measured values and their uncertainties; * physical constraints relating the measured quantities; * statistical inference methods.
Measured values are adjusted so that the reconciled values satisfy the physical laws while remaining consistent with the uncertainty associated with each measurement. The resulting reconciled values are statistically more likely than the original measurements and generally exhibit reduced uncertainty. [16_IMOC_paper_22 | PDF] Besides uncertainty reduction, data reconciliation can be used to: * detect inconsistent measurements; * identify possible instrumentation defects; * reveal modelling deficiencies.
Benefits of Performing Reconciliation in OpenModelica¶
- Implementing data reconciliation in OpenModelica offers several advantages:
The equations required for data reconciliation can be automatically extracted from an existing Modelica model;
The Modelica model does not need to be simulable (making it possible to perform reconciliation on a model that fails to initialize correctly).
Apart from this "equation extraction" phase — which is specific to OpenModelica — the current implementation of the "reconciliation" component follows the assumptions and calculation procedures specified in the VDI 2048 standard. Current assumptions are: - The extracted equations should represent a static model (model featuring algebraic equations only, no differential equation). - The uncertainties associated to the measured values follow a Gaussian distribution.
Pieces of Data Reconciliation Theory¶
Prior and Reconciled Distributions¶
The measurements are assumed to follow a multivariate Gaussian distribution characterised by: * a vector of measured values; * a variance-covariance matrix describing uncertainties and correlations.
The physical constraints extracted from the Modelica model define admissible operating states. Data reconciliation computes the conditional distribution satisfying these constraints.
The outputs of the reconciliation process are: * reconciled values; * reconciled uncertainties; * reconciled covariance matrix.
These quantities represent the best estimate of the system state under the assumptions provided by the model and measurement uncertainty characterisation.
Statistical Validation¶
The validity of reconciliation results must be assessed using statistical tests.
** Global Consistency Test **
A global test evaluates whether the corrections required to satisfy the physical constraints remain compatible with the uncertainty assumptions.
A failure of the global test indicates at least one inconsistency among:
* the measurements;
* the uncertainty model;
* the physical model
** Local Consistency Tests ** Local tests are performed for individual measurements. A reconciled value requiring a correction exceeding its expected uncertainty range may indicate: * a faulty sensor; * underestimated measurement uncertainty; * missing physical phenomena in the model.
Statistical tests should always be analysed before using reconciled values for operational or engineering decisions.
Recommended Workflow¶
- The complete data reconciliation workflow consists in four main phases:
Define the reconciliation problem;
Quantify measurement uncertainties;
Execute data reconciliation;
Analyze the results.
The above figure summarizes the workflow: light blue refers to steps implemented in OpenModelica while dark blue highlights steps performed in a third-party tool dedicated to uncertainty quantification.
Main Feature : Validate and Reconcile Measured Values¶
Step 1: Define the Data Reconciliation Problem in OpenModelica¶
- To perform data reconciliation, the user first identifies two kinds of variables:
The boundary conditions (which represent assumptions on the environment);
The variables of interest to be reconciled.
- These two kinds of variables must be manually declared in the Modelica source code
With annotations for the boundary conditions;
With modifiers for the variables of interest.
The boundary conditions are declared with the annotation: annotation(__OpenModelica_BoundaryCondition = true).
The variables of interest are declared with the modifier uncertain = Uncertainty.refine.
OpenModelica uses this information to automatically extract from the Modelica model the equations required for reconciliation.
Let us take the example of the Modelica model of a splitter.
Water flows from left to right, from the source to the sinks. The model is made of five different model components.
For the pressure source component, the boundary conditions are the pressure P0, and the specific enthalpy h0 or the temperature T0 of the source (depending on the option chosen by the user).
model SourceP "Water/steam source with fixed pressure"
parameter Modelica.SIunits.AbsolutePressure P0=300000 "Source pressure" annotation(__OpenModelica_BoundaryCondition = true);
parameter Modelica.SIunits.Temperature T0=290 "Source temperature (active if option_temperature=1)" annotation(__OpenModelica_BoundaryCondition = true);
parameter Modelica.SIunits.SpecificEnthalpy h0=100000
"Source specific enthalpy (active if option_temperature=2)" annotation(__OpenModelica_BoundaryCondition = true);
parameter Integer option_temperature=1 "1:temperature fixed - 2:specific enthalpy fixed";
Modelica.SIunits.AbsolutePressure P "Fluid pressure";
Modelica.SIunits.MassFlowRate Q "Mass flow rate";
Modelica.SIunits.Temperature T "Fluid temperature";
Modelica.SIunits.SpecificEnthalpy h "Fluid enthalpy";
equation
P = P0;
if (option_temperature == 1) then
T = T0;
h = f(T);
else
h = h0;
T = g(h);
end if;
end SourceP;
- Boundary conditions are declared with annotations so that libraries can be modified to accomodate data reconciliation,
and still can be used with tools that do not support data reconciliation (because annotations not recognized by a tool are ignored by that tool).
Variables of interest are however marked with modifiers instead of annotations so that checks can be performed to detect errors such as declaring a variable that does not exist to be a variable to be reconciled. The drawback is that tools that do not support data reconciliation will produce an error.
Hint
To avoid this problem, variables of interest should be declared in a separate model (Splitter_DR in the example below) that instantiates the orginal model (Splitter),
so that the original model (Splitter) is not modified and can still be used with tools that do not support data reconciliation.
model Splitter_DR
Splitter splitter(
pipe1(Q(uncertain = Uncertainty.refine)),
pipe2(Q(uncertain = Uncertainty.refine)),
pipe3(Q(uncertain = Uncertainty.refine)),
pipe1(Pm(uncertain = Uncertainty.refine)),
pipe2(Pm(uncertain = Uncertainty.refine)),
pipe3(Pm(uncertain = Uncertainty.refine)),
pipe1(T(uncertain = Uncertainty.refine)),
pipe2(T(uncertain = Uncertainty.refine)),
pipe3(T(uncertain = Uncertainty.refine)));
end Splitter_DR;
Step 2: Quantify Measurement Uncertainties¶
Once boundary conditions and variables of interest have been declared, measurements and their uncertainties are collected. Uncertainty estimates and correlations between measurements should be determined before launching the reconciliation process. External uncertainty quantification method and tools such as GUM and OpenTURNS may be used to construct covariance matrices and estimate correlations.
- In practice, when uncertainty charaterization has been done, the user should provide measurement data and correlation information through two input files:
The measurement input file (mandatory).
The correlation matrix input file (optional).
- The measurement input file is a
.csvfile with three columns: One column for the variable names [ident]
One column for measured values [positive floating point number]
One column for the half-width confidence intervals [positive floating point number].
The half-with confidence interval for variable is defined as
, where
is the standard deviation of
and
The header of the file is the row:
Variable Names; Measured Value-x; Half Width Confidence Interval
It is possible to insert comments with // at the beginning of a row:
// Measurement input file for the Splitter model.
Variable Name;Measured Value;Half Width Confidence Interval
splitter.pipe1.Q; 2.50; 0.196
splitter.pipe2.Q; 1.15; 0.196
splitter.pipe3.Q; 1.25; 0.196
splitter.pipe1.Pm; 6.1e5; 0.392e5
splitter.pipe2.Pm; 2.55e5; 0.392e5
splitter.pipe3.Pm; 2.45e5; 0.392e5
splitter.pipe1.T; 292; 1.96
splitter.pipe2.T; 386; 1.91
splitter.pipe3.T; 388; 1.91
The above file can be more easily visualized in matrix form:
The correlation matrix file is a .csv file that contains the off-diagonal lower triangular
correlation coefficients of the variables of interest:
The first row contains names of variables of interest [ident].
The first column contains names of variables of interest [ident].
The names in the first row and first column must be identical in the same order.
The first cell in the first row (which is also the first cell in the first column) must not be empty, but can contain any character string (except column separators).
The off-diagonal lower triangular matrix cells contain the correlation coefficients [positive or nul floating point number]. The correlation coefficients
are defined such that
where
and
are respectively the standard deviations of variables
and
, and
is the covariance matrix.
because
The upper triangular and diagonal cells are ignored because the correlation matrix is symmetric
, and its diagonal is
Only variables of interest with positive correlation coefficients must appear in the matrix. Unfilled cells are equal to zero. Variables of interest that do not appear in the matrix have correlation coefficients equal to zero. Therefore, if all correlation coefficients are equal to zero, the matrix can be empty and the correlation matrix file is not needed
The following correlation file is drawn from the VDI2048 standard example of a heat circuit of a steam turbine plant.
Sxy;mV;mHK;mSPLL;mSPL;mFDKELL;mFDKEL
mV
mHK
mSPLL
mSPL;;;0.39951
mFDKELL;;;0;0
mFDKEL;;;0;0;0.2
The above file can be more easily visualized in matrix form:
The variables mV and mHK could have been omitted because they do not have any positive correlation coefficients.
Hint
The quality of reconciliation results strongly depends on the quality of uncertainty information. For each measurement, uncertainties should be evaluated according to the principles of the Guide to the Expression of Uncertainty in Measurement (GUM). Typical uncertainty sources include: * sensor calibration; * acquisition chain accuracy; * environmental effects; * measurement methodology; * repeatability. When measurements are correlated, correlation coefficients should also be estimated and included in the covariance matrix. External uncertainty quantification tools such as OpenTURNS may be used to: * propagate elementary uncertainty sources; * construct covariance matrices; * estimate correlations between measured variables.
Step 3: Execute Data Reconciliation¶
OpenModelica automatically extracts the auxiliary equations linking the measured variables, constructs the reconciliation problem, computes reconciled values and uncertainties, performs statistical tests and generates a reconciliation report.
In practice the data reconciliation setup is done in OMEdit by:
Opening the Modelica model with the data reconciliation modifiers.
Selecting Data Reconciliation > Calculate Data Reconciliation.
Selecting the Data Reconciliation algorithm.
Filling the Data Reconciliation form with
The name of the Measurement Input File (mandatory);
The name of the Correlation Matrix Input file (optional). The default is the identity matrix (i.e. measurements are independent of each other);
The value of Epsilon (optional). The default value is
1.e-10. Epsilon is the stopping criteria of the data reconciliation numerical iterations.
Clicking on Save Settings to save the above settings in the Modelica model.
Clicking on Calculate to launch the calculation
The data reconciliation computation is performed in three main steps:
Static analysis is performed on the model to extract the equations that are necessary for data reconciliation. There are two groups of extracted equations:
The auxiliary conditions that constrain the variables of interest.
The intermediate equations that solve the intermediate variables from the variables of interest (this is the numeric way of eliminating the intermediate variables).
The auxiliary and intermediate equations are interchangeable: denoting r the number of auxiliary conditions, there are as many possibilities to construct the set of auxiliary conditions as to choose r equations among the set that contains both the auxiliary and the intermediate equations.
Any error in the posing of the data reconciliation problem is detected at this step. The possible errors are:
The number r of auxiliary equations is not strictly less than the number of variables of interest.
The number r of auxiliary equations is zero.
Both errors occur when there are too many boundary conditions related to the variables of interest. Variables of interest that are not involved in any of the auxiliary conditions or intermediate equations are not reconciled.
The model is simulated to compute the Jacobian matrices and eliminate numerically the intermediate variables. At this step, numerical simulation errors can occur, such as divisions by zero, non-convergence, etc. They can be corrected by improving the model or providing better start values.
The input files are read, the data reconciliation calculation is performed, and the results are displayed. Errors can occur:
If input files do not exist.
If there is a mismatch between the variables of interest declared in the model and in the input files:
All variables of interest declared in the model should be declared in the measurement input file and reciprocally.
All variables of interest declared in the correlation matrix file should be declared in the model (the converse is not true).
If a variable of interest has multiple entries in an input file.
If the first row and the first column are different in the correlation matrix file.
If the numerical cells of the matrices are not positive real numbers.
Step 4: Analyze the Results¶
The reconciliation report must be analysed to verify the validity of the assumptions made on the measurements, uncertainty model and physical model.
Successful statistical tests indicate that reconciled values provide a more reliable estimate of the actual state of the system and uncertainty reduction can be used for subsequent engineering analyses.
When inconsistencies are detected, measurements, uncertainty assumptions and the physical model should be reviewed.
Data reconciliation should therefore be considered as an iterative process supporting instrumentation validation, uncertainty improvement and model refinement.
In practice, OMEdit displays the results:
In an
.htmlfile with the title: Data Reconciliation Report. This file automatically pops up when the calculation is completed.In two
.csvoutput files:One that contains the reconciled values and the reconciled half-width confidence intervals.
The other that contains the reconciled covariance matrix.
- These files do not pop up automatically when the calculation is completed. The names of the files are respectively
| <working directory>\\<model name>\\< model name>_Outputs.csv- and
| <working directory>\\<model name>\\< model name>_Reconciled_Sx.csv,
where <model name> denotes the full name of the model, including its path. The name of the working directory can be read with the command Tools > Options.
The Data Reconciliation Report has three sections:
- Overview, with the following information:
Model file: name of the Modelica model file
Model name: name of the Modelica model
Model directory: name of the directory of the Modelica model file
Measurement input file: name of the measurement input file
Correlation matrix input file: name of the correlation matrix file (if any)
Generated: date and time of the generation of the data reconciliation report
- Analysis, with the following information:
Number of auxiliary conditions, denoted
in the sequel.
Number of variables to be reconciled.
Number of related boundary conditions: number of boundary conditions related to the variables to be reconciled. This number should be strictly less than the number of variables to be reconciled, otherwise the data reconciliation problem is ill-posed and data reconciliation cannot be performed.
Number of iterations to convergence: number of iterations of the data reconciliation numerical loop.
Final value of
: final value of the data reconciliation iteration loop. This value is smaller than epsilon when the iterations are completed (cf. below).
Epsilon: stopping criteria of the data reconciliation iteration loop. The recommended value by VDI 2048 is
1.e-10.Final value of the objective function
: is equal to
multiplied by
, where
is the number of auxiliary conditions.
Chi-square value: value of the chi-square distribution for
degrees of freedom and statistical certainty of probability of 95%.
Result of global test: true if
is less than the chi-square value, false otherwise. If false, the results for the reconciled values should be rejected because the vector of contradictions (i.e. the discrepancy between the measured values and the reconciled values) is too large.
Auxiliary conditions: set of the auxiliary conditions (i.e., the equations that constrain the variables of interest).
Intermediate equations: set of the intermediate equations (i.e., the equations that compute the intermediate variables from the variables of interest). This set can be empty if there are no intermediate variables.
Debug log: log of the numerical iteration loop.
- Results, which is a table with the following columns:
Variables to be Reconciled: the names of the variables of interest.
Initial Measured Values: the measured values entered in the measurement input file.
Reconciled Values: the reconciled values computed by the data reconciliation algorithm.
Initial Half-width Confidence Intervals: the half-width confidence intervals entered in the measurement input file.
Reconciled Half-width Confidence Intervals: the reconciled half-width confidence intervals computed by the data reconciliation algorithm.
Results of Local Tests: true if the values of local tests (cf. below) are less than the quantile of normal distribution with probability 95% (
), false otherwise.
Values of Local Tests: values of the improvements (i.e., the difference between the initial and the reconciled values) divided by the square root of the diagonal element of the covariance matrix of the improvements.
Margin to Correctness:
minus the values of local tests.
The data reconciliation report for the the VDI2048 standard example of a heat circuit of a steam turbine plant is given below.
Overview:
---------
Model file: VDI2048Example.mo
Model name: NewDataReconciliationSimpleTests.VDI2048Example
Model directory: NewDataReconciliationSimpleTests
Measurement input file: VDI2048Example_Inputs.csv
Correlation matrix input file: VDI2048Example_Correlation.csv
Generated: Thu Jan 20 18:41:45 2022by OpenModelica v1.19.0-dev-500-g6c3a4e429f (64-bit)
Analysis:
---------
Number of auxiliary conditions: 3
Number of variables to be reconciled: 11
Number of related boundary conditions: 0
Number of iterations to convergence: 2
Final value of (J*/r) : 4.56744e-28
Epsilon : 1e-10
Final value of the objective function (J*) : 1.37023e-27
Chi-square value : 7.81473
Result of global test : TRUE
Auxiliary conditions
--------------------
1. (1): 0.0 = mHDANZ - mHDNK
2. (1): mFD3 = mHK + mA7 + mA6 + mA5 + 0.4 * mV
3. (1): mFD1 = mFDKEL + mFDKELL + (-0.2) * mV
Intermediate equations
----------------------
1. (1): mHDANZ = mA7 + mA6 + mA5
2. (1): mFD2 = mSPL + mSPLL + (-0.6) * mV
3. (1): 0.0 = mFD2 - mFD3
4. (1): 0.0 = mFD1 - mFD2
Debug log
---------
Results
-------
mD is not reconciled because it does not appear in any of the auxiliary conditions or intermediate equations.
- For the VDI2048 example, the name of the
.csvoutput file is | <working directory>\\NewDataReconciliationSimpleTests.VDI2048Example\\NewDataReconciliationSimpleTests.VDI2048Example_Outputs.csv- and the name of the reconciled covariance matrix csv file is
| <working directory>\\NewDataReconciliationSimpleTests.VDI2048Example\\NewDataReconciliationSimpleTests.VDI2048Example_ Reconciled_Sx.csv
The Modelica model of the VDI2048 example is given below.
model VDI2048Example
Real mFDKEL(uncertain=Uncertainty.refine)=46.241;
Real mFDKELL(uncertain=Uncertainty.refine)=45.668;
Real mSPL(uncertain=Uncertainty.refine)=44.575;
Real mSPLL(uncertain=Uncertainty.refine)=44.319;
Real mV(uncertain=Uncertainty.refine);
Real mHK(uncertain=Uncertainty.refine)=69.978;
Real mA7(uncertain=Uncertainty.refine)=10.364;
Real mA6(uncertain=Uncertainty.refine)=3.744;
Real mA5(uncertain=Uncertainty.refine);
Real mHDNK(uncertain=Uncertainty.refine);
Real mD(uncertain=Uncertainty.refine)=2.092;
Real mFD1;
Real mFD2;
Real mFD3;
Real mHDANZ;
equation
mFD1 = mFDKEL + mFDKELL - 0.2*mV;
mFD2 = mSPL + mSPLL - 0.6*mV;
mFD3 = mHK + mA7 + mA6 + mA5 + 0.4*mV;
mHDANZ = mA7 + mA6 + mA5;
0 = mFD1 - mFD2;
0 = mFD2 - mFD3;
0 = mHDANZ - mHDNK;
end VDI2048Example;
Note that the binding equations that assign fixed values to the variables have been automatically eliminated by the extraction algorithm. These equations are necessary to have a valid square Modelica model for simulation purpose, but must be eliminated for data reconciliation.
The table below compares the results obtained with OpenModelica with those given by the VDI2048 standard.
The value of mD is left unchanged after reconciliation. This is indicated by 'Not reconciled' in OpenModelica,
and by repeating the initial measured values in VDI2048. In order to compute the reconciled values of mFD1, mFD2, mFD3 and mHDANZ, which have no measurements, it is possible to consider them as variables of interest with very large half-width confidence, e.g., 1e4, and assign them arbitrary measured values, e.g. 0. The result is
For the splitter, the results are the following:
Hint
When inconsistencies are detected, several causes should be investigated: - Measurement Issues. Possible causes include: sensor drift, calibration errors, acquisition chain failures, incorrect measured values. - Uncertainty Characterisation Issues. Possible causes include: underestimated uncertainties, neglected correlations, incorrect uncertainty propagation assumptions. - Model Issues. Possible causes include: missing physical phenomena, invalid operating assumptions, modelling errors.
Do not forget that the reconciliation process is therefore iterative: discrepancies revealed by statistical tests should be used to improve either the measurements, the uncertainty model, or the physical model until a statistically consistent solution is obtained.
Additional Feature: Compute the Boundary Conditions from the Reconciled Values¶
- The values and uncertainties of the boundary conditions that correspond to the reconciled values of the variables of interest can be computed by:
Selecting Data Reconciliation > Calculate Data Reconciliation
Selecting the Boundary Conditions algorithm.
- Filling the Data Reconciliation form with
The name of the Reconciled Measurement File (mandatory). It is the name of the
.csvoutput file produced by the data reconciliation algorithm.The name of the Reconciled Correlation Matrix file (mandatory). It is the name of the correlation matrix
.csvfile produced by the data reconciliation algorithm.
Clicking on Save Settings to save the above settings in the Modelica model.
Clicking on Calculate to launch the calculation.
- At least one boundary condition must be declared in the model, otherwise the computation fails with a difficult to interpret error message such as
- Cannot Compute Jacobian Matrix
F.
A better error message will be posted in a future version.
- The computation of the boundary conditions is performed in three main steps:
- Static analysis is performed on the model to extract the equations that compute the boundary conditions from the variables of interest. This corresponds to automatically inverting the original model. There are two groups of extracted equations:
The boundary conditions that are the equations that compute the boundary conditions.
The intermediate equations that solve the intermediate variables from the variables of interest (this is the numeric way of eliminating the intermediate variables).
2. The model is simulated to compute the Jacobian matrices and eliminate numerically the intermediate equations. At this step, numerical simulation errors can occur, such as divisions by zero, non-convergence, etc. They can be corrected by improving the model or providing better start values.
The input files are read, the numerical calculations are performed, and the results are displayed. The half-width confidence intervals for the boundary conditions are calculated by propagating the reconciled uncertainties on the variables of interest through the inverted model.
- The results are displayed:
In an
.htmlfile with the title: Boundary Condition Report. This file pops up automatically when the calculation is completed.
2. In a
.csvoutput file. The name of the.csvoutput file is<working directory>\\<model name>\\< model name>_ BoundaryConditions_Outputs.csv
The Boundary Condition Report has three sections:
Overview
--------
a. Model file: name of the Modelica model file
b. Model name: name of the Modelica model
c. Model directory: name of the directory of the Modelica model file
d. Reconciled values input file: name of the csv output file produced by the data reconciliation algorithm
e. Reconciled covariance matrix input file: name of the correlation matrix csv file produced by the data reconciliation algorithm
f. Generated: date and time of the generation of the boundary condition report
Analysis
--------
a. Number of boundary conditions.
b. Number of reconciled variables.
c. Boundary conditions: set of the boundary conditions (i.e., the equations that compute the boundary conditions).
d. Intermediate equations: set of the intermediate equations (i.e., the equations that compute the intermediate variables from the variables of interest).
Debug log
---------
Log of the numerical iteration loop
Results
-------
A table with the following columns
a. Boundary Conditions: the names of the boundary conditions.
b. Values: the computed values of the boundary conditions.
c . Reconciled Half-width Confidence Intervals: the half-with confidence intervals for the boundary conditions.
The results for the splitter are:
Contacts¶
References¶
Bouskela, D., Jardin, A., Palanisamy, A., Ochel, L., & Pop, A. (2021). New Method to Perform Data Reconciliation with OpenModelica and ThermoSysPro. Proceedings of 14th Modelica Conference 2021, Linköping, Sweden, September 20-24, 2021.
Mazurié, B., Jardin, A., Borel, P., Davelaar, F., Boldo, D., Corona Mesa‑Moles, L. (2025). Data Reconciliation for Industrial Experiments. Proceedings of the 16th International Modelica & FMI Conference, Lucerne, Switzerland, September 8-10, 2025.
VDI - Verein Deutscher Ingenieure. (2000). Uncertainty of measurement during acceptance tests on energy-conversion and power plants - Part 1: Fundamentals. VDI 2048 Blatt 1, October 2000.
GUM - ISO/IEC GUIDE 98-3, Uncertainty of measurement - Part 3: Guide to the expression of uncertainty in measurement, 1995.
OpenTURNS: An Open source initiative for the Treatment of Uncertainties, Risks'N Statistics}, https://openturns.github.io/.
