
    JOSEPH A. and Josephine A., by their next friend, Corrine WOLFE; Michael B., by his next friend Dr. Lucy Gale McMurray; Michelle C., by her next friends La Donna Harris and Dr. Lucy Gale McMurray; Joel D., by his next friend Dr. Lucy Gale McMurray; Susan E. and Donald E., by their next friend Barbara Burns, on their own behalf and on behalf of all others similarly situated, Plaintiffs-Appellants, v. NEW MEXICO DEPARTMENT OF HUMAN SERVICES; Lawrence B. Ingram, individually and as Secretary of the New Mexico Department of Human Services; Margaret Larragoite, individually and as Director of the Social Services Division of the New Mexico Department of Human Services; Darold Christiansen, individually and as Director of the Bernalillo County Social Services Division of the Department of Human Services; Janet Bryan, individually and as Director of the Santa Fe County Social Services Division of the Department of Human Services, Defendants-Appellees.
    No. 94-2160.
    United States Court of Appeals, Tenth Circuit.
    Nov. 9, 1995.
    Rehearing Denied Dec. 14, 1995.
    
      Marcia Robinson Lowry of the Children’s Rights Project, American Civil Liberties Union, New York City (Susan Lambíase and C. Eduardo Rodriguez of the Children’s Rights Project, and Robert Levy, Levy & Greer, Albuquerque, N.M., with her, on the briefs), for Plaintiffs-Appellants.
    Robyn B. Hoffman, Assistant Attorney General, Office of the Attorney General, Albuquerque, N.M. (Tom Udall, Attorney General, and Robert Tabor Booms, Office of the Attorney General, and Stevan Douglas Looney, Crider, Calvert & Bingham, Albuquerque, N.M., with her on the brief), for Defendants-Appellees.
    Before SEYMOUR, Chief Judge, MOORE, and EBEL, Circuit Judges.
   EBEL, Circuit Judge.

This case involves the dissolution of a consent decree governing the New Mexico foster care system. The decree requires that the State of New Mexico’s Department of Human Services (“the Department”) and its top officials (collectively “Defendants”) fulfill certain goals and specific requirements. Furthermore, the decree, spawned by a class action brought by a number of children in the state’s foster care system (“Plaintiffs”), provides that it may be terminated when Defendants maintain “substantial and continuous compliance” for twelve consecutive months. The district court dissolved the decree based on a Special Master’s report that the state had met this requirement. Plaintiffs appealed. However, we are unable to evaluate the claim that the Special Master incorrectly defined and applied the “substantial compliance” standard because the Special Master’s factual findings and conclusions of law do not conform to Fed.R.Civ.P. 52(a). Therefore, we VACATE the judgment of the district court and REMAND this case for further proceedings.

I. BACKGROUND

Because we dispose of this ease on Rule 52(a) grounds, we only outline the basic facts and history concerning the consent decree. Plaintiffs are a certified class of neglected and dependent children under the care and in the custody of the Department. Judge Juan Burciaga certified Plaintiffs as a class as a result of a 1980 suit brought by the ACLU Children’s Rights Project which challenged the state’s foster care system on the ground that the Department failed to develop and implement adequate plans for the permanent placement of children within its custody. On September 23,1983, Judge Burcia-ga approved a consent decree providing for long-term and wide-ranging reform of the state’s foster care system to ensure that homeless, neglected or otherwise dependent children would not languish in the state’s care. The terms of that decree are attached to Judge Burciaga’s opinion in Joseph A. v. New Mexico Dep’t of Human Servs., 575 F.Supp. 346 (D.N.M.1982).

In 1988, Defendants filed with the district court a Motion for Judicial Review and an Order of Compliance requesting that the court find that they had substantially complied with the provisions of the decree for twelve continuous months so as to require its dissolution. See Joseph A., 575 F.Supp. at 364. As a result of this motion, and because of Plaintiffs’ motion for contempt, the special master, Frank Zinn (“the Special Master”), held hearings to resolve these outstanding issues. In June, 1989, the Special Master reported that Defendants had substantially complied with the decree for a continuous period of twelve months and recommended that the district court terminate the decree. Judge Burciaga rejected the findings in the Special Master’s report as inadequate to terminate the decree, explaining that “[t]he court must be satisfied that the spirit and intent of the Consent Decree will continue to be served beyond termination of the Consent Decree.” quoted at Aplt.Br. at 4. The court then ordered the Special Master to hold further hearings and make additional findings of fact and conclusions of law to address the court’s concerns.

On April 30, 1993, after holding additional hearings, the Special Master issued a further report recommending that the court dissolve the consent decree because the Department had been in substantial compliance with the decree for twelve continuous months. In this report, the Special Master considered both expert opinions and statistical evidence in determining that Defendants were adequately complying with the terms of the decree. The Special Master organized the requirements of the decree and his report into ten distinct areas: (1) training; (2) caseloads; (3) planning & review; (4) adoptions; (5) legal services; (6) information & records; (7) staff qualifications; (8) citizens review board; (9) general compliance; and (10) future comportment. The Special Master’s report contained three main sections: (1) findings of fact; (2) summary of evidence; and (3) discussion. At the end of all but two of the findings of fact made in each of the ten areas, the Special Master concluded that Defendants were in substantial and continuous compliance with the specific requirements of the decree. Based on these findings, the Special Master recommended that the court terminate the decree.

On September 14, 1993, Judge Burciaga affirmed the Special Master’s 1989 and 1993 reports. Plaintiffs then filed a motion for reconsideration, and Defendants filed a motion requesting the termination of the consent decree. Plaintiffs specifically highlighted the alleged unreliability of the Department’s computer record keeping system (the ADAPT measurement system), relied upon by the Special Master, as well as the low levels of compliance in certain areas. Moreover, Plaintiffs also criticized the Special Master’s conclusion that case studies of several children who languished under the state’s care constituted “selected instances” which should not influence his finding of “substantial compliance.” See Aplt.Br. at 20. On June 30, 1994, the district court denied Plaintiffs’ motion for reconsideration and terminated the consent decree. Plaintiffs now appeal the district court’s order denying the motion for reconsideration and terminating the decree, as well as the court’s order adopting the reports of the Special Master.

II. DISCUSSION

In order to review whether the dissolution of the consent decree was proper, we must (1) define the “substantial compliance” standard and (2) then consider whether the Special Master (and district court) correctly applied that standard to the facts of the instant case. However, because the findings of the Special Master lack the requisite degree of specificity and clarity, we are unable to complete the second step of our review.

A. The “Substantial Compliance” Standard

The term “substantial compliance” appears in the consent decree and thus we treat it like a contract term. See United States v. ITT Continental Baking Co., 420 U.S. 223, 236-38, 95 S.Ct. 926, 934-35, 43 L.Ed.2d 148 (1975) (explaining that consent decrees “should be basically construed as contracts”). Accordingly, we review the district court’s definition of that term de novo. North Shore Lab. v. Cohen, 721 F.2d 514, 518-19 (5th Cir.1983) (interpretation of a consent decree, like that of a contractual provision, requires de novo review). See NLRB v. Monfort, Inc., 29 F.3d 525, 528 (10th Cir.1994) (reviewing special master’s conclusions of law de novo).

The phrase “substantial compliance” is not susceptible of a mathematically precise definition. In interpreting a similar phrase in a consent decree, the First Circuit in Fortin v. Commissioner of the Dep’t of Mass. Pub. Welfare, 692 F.2d 790, 795 (1st Cir.1982) observed that:

no particular percentage of compliance can be a safe-harbor figure, transferable from one context to another. Like “reasonableness,” “substantiality” must depend on the circumstances of each ease, including the nature of the interest at stake and the degree to which noncompliance affects that interest.

692 F.2d 790, 795 (1st Cir.1982) (internal citation omitted). See also Ruiz v. McCotter, 661 F.Supp. 112, 147 (S.D.Tex.1986) (citing Fortin and comparing concept to “reasonableness”).

We have explained that the contract law doctrine of substantial compliance is:

simply a doctrine to assist the court in determining whether conduct should, in reality, be considered the equivalent of compliance under the contract. See John D. Calamari & Joseph M. Perillo, The Law of Contracts § 11-15, at 454 (3d ed. 1987) (“If a party has substantially performed, it follows that any breach he may have committed is immaterial.”).

Peckham v. Gem State Mut., 964 F.2d 1043, 1052 (10th Cir.1992). Judge Cardozo, in the seminal substantial compliance case of Jacob & Youngs, Inc. v. Kent, concluded that performance of a contract will not be considered in substantial compliance of the contract if the deviation from the contract requirements ... “in any real substantial measure ... frustrated the purpose of the contract.” 230 N.Y. 239, 129 N.E. 889, 891 (1921). Thus, the touchstone of the substantial compliance inquiry is whether Defendants frustrated the purpose of the consent decree— i.e. its essential requirements.

The consent decree’s preamble clearly states that its essential purposes are to establish:

... fair, reasonable and timely decision making with regard to access of adoption, and ... fair, reasonable and adequate procedures and practices necessary to insure access to permanent adoptive homes ...

Joseph A., 575 F.Supp. at 354. To accomplish this purpose, the consent decree endeavored to:

put in place a system, equivalent to regulations, for dealing with children in DHS [Department] custody. It requires planning, action, supervision, and oversight on a sequential timetable from first intake to final discharge.

Sp.Mast. at 3. Thus, substantial compliance with the decree contemplates more than a one-time 12 month period of “fair, reasonable, and timely decision making” with regard to adoption. Substantial compliance with the decree also requires enactment of “fair, reasonable, and adequate procedures” to “effect long-term institutional reform of the New Mexico foster-care system.” Id. (quoting Burciaga, J.). Cf. Board of Education of Oklahoma City Public Schools v. Dowell, 498 U.S. 237, 111 S.Ct. 630, 112 L.Ed.2d 715 (1991) (reviewing continued enforcement of a school desegregation decree).

The Special Master correctly noted that the “substantial compliance” analysis needed to focus on both the purposes of the decree— that is, the (1) timely; (2) fair; and (3) reasonable process of adopting foster care children — as well as the institutionalization of procedures to accomplish those ends. Sp. Mast, at 48. However, the Special Master cast some doubt on the legal standard he was using when he explained that “substantial compliance” reflected “no absolutes” and that “the Court must subjectively weigh what was done against what was wanted.” Id. The Special Master then referred to a circular definition of substantial compliance when he relied on the Webster’s New Collegiate Dictionary (1961) which defined “substantial” as “that is such in substance or in the main, as, a substantial victory,” or “considerable; large; as a substantial gain.” Id.

This dictionary definition is most unhelpful and it suggests a degree of imprecision and latitude that goes beyond the legal concept of “substantial compliance” or “substantial performance.” On remand, the court should begin with the essential purposes of the consent decree which we have quoted from the Decree’s preamble, and it should then consider the specific steps set forth in the consent decree by which those purposes may be satisfied. To the extent that any stipulated criteria has not been met, the court must determine whether that failure is immaterial to the overall objectives or, on the other hand, whether it had a material adverse impact upon the overall processing and placement of children into permanent homes. Because the consent decree sets forth specific criteria to be met, those criteria must be respected unless a deviation can be shown not to have a material effect upon the overall performance of the Department in processing and placing children into permanent homes. This kind of analysis is not reflected in either the Special Master’s report or the district court opinion below.

B. Rule 52(a) Findings

Plaintiffs argue that, even though the Special Master’s lengthy report included a summary of the evidence, it failed to make specific, non-conclusory factual findings in accordance with the requirements of Rule 52(a). Rule 52(a) provides that a district court “shall find the facts specially and state separately its conclusions of law thereon.” This rule serves to (1) engender care on the part of trial judges in ascertaining the facts; and (2) make possible meaningful appellate review. Ramey Constr. Co., Inc. v. Apache Tribe, 616 F.2d 464, 466-67 (10th Cir.1980). Thus, the touchstone for whether findings of fact satisfy Rule 52(a) is whether they are “sufficient to indicate the factual basis for the court’s general conclusion as to ultimate facts” so as to facilitate a “meaningful review” of the issues presented. Otero v. Mesa County Valley Sch. Dist., 568 F.2d 1312, 1316 (10th Cir.1977). If a district court fails to meet this standard — i.e. making only general, eonclusory or inexact findings — we must vacate the judgment and remand the ease for proper findings. Battle v. Anderson, 788 F.2d 1421, 1425 (10th Cir.1986). See Roberts v. Metropolitan Life Ins. Co., 808 F.2d 1387, 1390-91 (10th Cir.1987).

[In order to satisfy Rule 52(a) ] the trial court must include as many of the subsidiary facts as necessary to permit us to determine the steps by which [it] reached its ultimate conclusion_ Where the trial court provides only eonclusory findings, unsupported by subsidiary findings or by an explanation of the court’s reasoning with respect to the relevant facts, a reviewing court simply is unable to determine whether or not those findings are clearly erroneous, [internal citation and quotation omitted.]

In the instant case, we cannot meaningfully review the lower court’s legal and factual rulings because the Special Master’s report (1) does not explicitly conclude that Defendants complied with the essence of each of the decree’s overall objectives nor does it address specifically why the failure to meet certain of the enumerated criteria had no material adverse impact upon the overall satisfaction of the consent decree’s objectives; (2) it fails to explain the methodology used; and (3) it fails to explain or set forth the essential subsidiary facts necessary to support its conclusions concerning Defendants’ substantial compliance with the decree’s specific requirements.

1. THE GOALS OF THE DECREE

In order to comply substantially with the decree, Defendants needed to satisfy the essential requirements of each of the basic goals of the decree — i.e. the implementation of a (1) timely, (2) fair, and (3) reasonable adoption process. The consent decree then sets forth specific criteria and procedures whereby those overall goals may be met. The Special Master’s report failed to conclude whether Defendants complied substantially with each of these essential purposes of the consent decree and it failed adequately to address those instances where the subsidiary criteria were not met. First, the Special Master’s report failed to address at all whether the decision making was fair and reasonable. Second, while the report discussed the timeliness of the “decision making with regard to access to adoption,” Sp.Mast. at 48, it failed adequately to address whether Defendants satisfied this mandate of the decree. For example, the report noted that Dr. Stein found that the overall length of time that children spent actually in custody before permanent placement increased from 1.5 to 2.1 years from 1988 to 1991, but the report failed adequately either to explain or discredit that testimony.

2. METHODOLOGY

The Special Master’s report also failed to outline his methodology for considering whether Defendants satisfied their obligations under the decree. Plaintiffs suggest that the Special Master erred both by ignoring the qualitative case studies and by crediting the unreliable quantitative evidence provided by the ADAPT system. We cannot resolve these objections, however, because the Special Master’s report faded to address them or explain the methodology used. For example, the Special Master discounted— without any explanation — the individual case studies, admitting that they “turned out badly,” but explaining that “[m]y findings are not influenced by these selected instances.” Sp.Mast. at 49. While these case studies do not necessarily render Defendants’ performance deficient, the Special Master should have explained why they were “selective” and aberrational rather than indicative of broader problems in the system as a whole.

3. THE SPECIFIC REQUIREMENTS

The Special Master’s report also often failed to explain the apparent discrepancy between a low compliance figure and his conclusion that, notwithstanding that data, Defendants had substantially complied with a specific requirement of the decree. For example, the Special Master noted that district lows and highs for the timeliness and content of the 30-day conferences ranged from 6.2% to 63%, while other testimony concluded that the plans satisfied at least the content requirement from 31% to 97% of the time. The Special Master then goes on without further explanation to conclude that “[t]here is a high occurrence level for this event” and that Defendants had substantially complied with this requirement. Sp.Mast. at 52-53. Another such example is the Special Master’s treatment of the requirement that the children’s attorney be notified every month as to whether they were freed for adoption. The report indicated that only 26% of the attorneys for the children in the state’s care were notified on time and 8% of the attorneys were never notified after establishing a plan for adoption; nonetheless, without further explanation except for a reference to caseworker attorney contacts (which were often also untimely), and a conclusion that late notification did not lead to “cessation of movement,” the report concluded that Defendants had substantially complied with this requirement. Sp.Mast. at 55-56. These conclusions of substantial compliance are non-sequiturs based on the facts stated in the report. These are but two examples, but similar leaps of faith or non-sequitur conclusions abound in the report. Thus, we are simply unable to review the district court’s rulings that Defendants substantially complied with the specific terms of the decree. On remand, the court must outline the essential subsidiary findings and reasoning which lead to its ultimate conclusions as to each specific requirement.

The Defendants in their brief argue that they presented opposing evidence to explain or refute the Plaintiffs’ evidence, but it is not the role of the Court of Appeals to weigh such evidence and make evidentiary findings as to what evidence is most credible. It is the role of the district courts to engage in clear and specific factfinding, and we are neither equipped nor inclined to assume that role. Moreover, given the length of the record in this case, it is especially inappropriate that we study the record in an attempt to discern whether the district court could have found the facts necessary to justify its holding. See Hull v. United States, 971 F.2d 1499, 1513 (10th Cir.1992), cert. denied, — U.S. -, 113 S.Ct. 1844, 123 L.Ed.2d 469 (1993) (even though support exists in record for district court’s judgment, the insufficiency of district court’s findings requires remanding the case to the district court). Thus, we VACATE the judgment of the district court and REMAND this case for further proceedings. Id.; Battle, 788 F.2d at 1425.

III. CONCLUSION

Because the Special Master’s report (and district court’s order) failed to make the specific and clear findings required by Rule 52(a), we are unable to determine whether the court correctly applied the substantial compliance standard. Thus, we VACATE the judgment of the district court and REMAND this case for further proceedings. On remand, the district court should (1) apply the substantial compliance standard set forth in this opinion — i.e. consider whether Defendants satisfied the essential purposes of the decree, which includes fair, reasonable, and timely decision making and fair, reasonable, and adequate procedures and practices necessary to insure access to permanent placement; (2) outline its reasoning why Defendants satisfied (or did not satisfy) the essential purposes of the decree — i.e. justify its ultimate legal conclusions. If the Defendants are found to have deviated from any of the enumerated criteria set forth in the consent decree, the court should explicitly address whether or not such deviation materially interfered with the essential purposes of the consent decree; (3) explain the methodology employed to reach its factual findings, and address the adequacy or inadequacy of the ADAPT data and episodal examples of noncomplianee; and (4) specify the subsidiary factual findings and reasoning necessary to support its judgment as to Defendants’ compliance with each of the overall goals and enumerated criteria in the decree. 
      
      . We deny Appellants’ motion to file a supplemental study with this court. The study in question was not part of the record below. Although we decline to award sanctions in this case as requested by Appellees, we expressly disapprove of Appellants’ submission of this motion and attached study to us.
     
      
      .Plaintiffs submitted a State Auditor’s report calling ADAPT’s reliability into question. However, because we confine our review to the findings and conclusions of the Special Master and the Special Master did not consider that report, the State Auditor's report is not properly before us, and thus, we do not consider it. Moreover, because we confine our analysis to that information considered by the Special Master, we reject Appellants’ motion to supplement the record with a recent study by the Children, Youth and Families Division of the New Mexico Social Services Division.
     
      
      .We reject Defendants' contention that Plaintiffs have failed properly to appeal the district court’s order because the district court adopted both the 1989 report as well as the 1993 report, and Plaintiffs have only appealed the Special Master's 1993 report. That is, Defendants argue that Plaintiffs only appealed half of the district court's order, and that Plaintiffs have waived any objections to the findings made in the 1989 report as incorporated into the court's 1994 order finding substantial compliance. The district court, however, explained that the 1989 report, taken alone, was inadequate to terminate the consent decree, and the district court's ultimate order considered both the 1989 and 1993 reports. Plaintiffs’ appeal of the district court’s order puts both reports at issue.
     
      
      .We note that this doctrine is also often referred to as the “substantial performance” doctrine. However, for simplicity purposes, this opinion will refer to the doctrine solely as the "substantial compliance” doctrine.
     
      
      . The district court underscored that the finding of substantial compliance must evidence that "the spirit and intent of the Consent Decree will continue beyond termination.” Sp.Mast. at 62.
     
      
      . Rule 52(a) also applies to findings made by a special master “to the extent that the court adopts them.”
     
      
      . Courts and commentators have consistently underscored this requirement. See Kelley v. Everglades Drainage Dist., 319 U.S. 415, 422, 63 S.Ct. 1141, 1145, 87 L.Ed. 1485 (1943) ('TChere must be findings, stated either in the court's opinion or separately, which are sufficient to indicate the factual basis for the ultimate conclusion."); Snyder v. United States, 674 F.2d 1359, 1363 (10th Cir.1982) (trial court must specify all subsidiary facts necessary "to afford appellate court a clear understanding of the ground or basis of the decision”); 9A Charles A. Wright & Arthur R. Miller, Federal Practice and Procedure, § 2579, at 539 (1995) (lower courts should specify as many of the "subsidiary facts as necessary to disclose to the reviewing court the steps by which the trial court reached its ultimate conclusion on each factual issue”).
     
      
      . While the Special Master's "Findings of Facts” Section appears to state his "Conclusions of Law," the labelling of the report does not inform or alter our Rule 52(a) analysis.
     
      
      . The Special Master erroneously concluded that, while each party asked for “findings on basic factual issues and also on many evidentiary details,” he believed that "findings on details are not necessary” because the report only needed to "decide and declare the ultimate facts of compliance with each component and overall compliance.” Sp.Mast. at 15. As explained in the text, findings on details are necessary to the extent that those details are logically required to support the ultimate facts and legal conclusions. That is, merely deciding and declaring the ultimate facts fails to satisfy Rule 52(a) and prevents us from engaging in any meaningful appellate review.
     
      
      . This failure to focus on the fairness and reasonableness of the foster care system might stem from Plaintiffs’ implicit concession that the consent decree's requirements, if substantially complied with, would necessarily accomplish these goals. That is, Plaintiffs might agree that the decree's requirements (e.g., the caseload limits and planning conferences), if put into practice, would ensure that the adoption process would not be arbitrary and capricious. In that case, the Special Master should have explicitly articulated this rationale (or another rationale) for not focusing on the goals of fairness and reasonableness.
     
      
      . As we noted earlier, the State Auditor's report criticizing the performance of the ADAPT system is not properly before us. However, Plaintiffs offered other evidence (e.g. Quality Assurance Data, App. A. 1296-98, 1311-14, 1338-43, 1374-76) which suggested that ADAPT’s data did not accurately characterize Defendants' performance and that should have been addressed by the Special Master.
     
      
      . This is not the first occasion in this litigation in which we have remanded a ruling by the district court because it was premised on insufficient findings. Indeed, in a related appeal involving attorney’s fees, we held that the district court's judgment suffered from the "analytical deficiencly]” of failing to "adequately explain the relationship between the need for the work [i.e. its legal basis] and the fees awarded.” Joseph A. v. New Mexico Dep't of Human Servs., 28 F.3d 1056, 1061 (10th Cir.1994). We remanded the case for farther findings so that we could engage in a meaningful appellate review.
     