With two equations, plus the equation relating LMTD with the temperatures, you can solve three variables, the others should be known by other ways. I understand that you know U,A,Tc and T_in, mdot, and Cp, so the solution should be immediate. Why the iterations? The Modelica solver will take care of founding the solution. The only possible problem I see is with the LMTD that some times goes to indefined mathematical operations. This can be solved by specifying altenative calculations when, for example, inlet and outlet temperature are the same.