Why Fire Tests Flunked

The ruling by a Federal District Judge in Brooklyn that the city's 1999 and 2002 exams for Firefighter discriminated against minority candidates creates a major headache for the Fire Department. Most painful of all is that it is a headache of the city's making.

It might seem that Judge Nicholas Garaufis is coming down on the opposite side of the issue from the U.S. Supreme Court, which a month ago found that the City of New Haven discriminated against 17 whites and one Latino seeking promotions in their Fire Department when it threw out the exams because of fear of litigation by minority candidates.

In fact, however, there is a consistency between the two decisions, each of which applied similar standards. In the New Haven case, it was decided by the majority of the High Court that there was nothing, other than the disparate impact shown in the results by race, to suggest the exams were flawed. The two city Firefighter tests, on the other hand, contained enough flaws to drive a fire engine through.

In each instance, there is a single moment of unassailable logic that makes the judicial ruling not only clear but obvious.

In the New Haven case, Supreme Court Justice Anthony Kennedy pointed out that when the exams came under fire, the consultant who prepared them asked to have a study done to verify whether they were fair and job-related. New Haven officials turned down that request, suggesting they were concerned with other matters besides whether the tests were valid. That conclusion amounted to game, set and match in Justice Kennedy's opinion on behalf of the court's majority.

For the city's 1999 exam for Firefighter, which in many ways served as the model for the 2002 test that was also challenged by the U.S. Justice Department after a complaint was brought by the FDNY Vulcan Society, the consultant interviewed two different groups totaling 16 firefighters to determine the key abilities needed to perform the job.

This produced a list of 18 abilities, but only nine of them were tested for on the 1999 exam. What Judge Garaufis found most astonishing about the exclusion of half the key components from the exam structure was that, "The two abilities which incumbent firefighters rated highest in importance—Oral Comprehension and Oral Expression—were not among those tested . . ."

The explanation given by city officials was that it wouldn't have been feasible to conducted "structured interviews" with thousands of candidates in order to determine their oral abilities. The Department of Citywide Administrative Services official who devised the 1999 exam testified that he didn't "do anything specific" to determine whether those two abilities or the seven others were "testable," saying he had simply followed "standard operating procedure."

"Standard operating procedure" for a test in an agency whose uniformed firefighter force was more than 90 percent white at the time the exams were held amounted to an invitation to have the city's defense laughed out of court. Judge Garaufis essentially did so.

The problem with the omission of any form of oral testing is glaringly evident. In order to overcome a finding of disparate racial impact, an exam must be proven to be job-related and that its use is a matter of business necessity in determining which candidates are most likely to perform a job capably. It's hard to argue that the fire tests were accomplishing that purpose if they didn't attempt to assess candidates' abilities in the two areas that working firefighters had deemed the most important.

This was hardly the only problem with the two exams. Judge Garaufis noted that while firefighters had helped devise some of the questions, no testing experts were deployed to refine those questions to ensure that there wasn't a bias to any of them.

The exams also were deemed by Judge Garaufis to demand too high a reading level. In the New Haven case, he noted, the reading level was set "below the tenth grade," and that was for promotion tests; a testing expert testified in the city case that the reading level for the two Firefighter exams was "above the 12th grade and 'was too high for the job of Firefighter.' ''

Judge Garaufis also questioned the cutoff scores that were deemed passing, in particular the use of a mark slightly above 84 for the 1999 test. City officials said those cutoffs were set based on the Fire Department's expected hiring needs. But he deemed that to be an improper reason for denying those who scored slightly lower a chance at the job. This was particularly problematic, he said, because those whose written marks were below the cutoff were denied the opportunity to take the physical test, where they might have scored high enough to place them above other candidates who made the cutoff.

Uniformed Firefighters Association President Steve Cassidy is among those who believes the city has wrongly de-emphasized the physical in recent years and overvalued the written test. Vulcan Society President John Coombs believes the written test should be scrapped altogether.

That, we believe, would throw the baby out with the bathwater. But clearly the FDNY must rethink future testing.

The irony is that the 2007 exam, aided greatly by a more-intensive recruitment campaign in the minority community, produced results more reflective of the city's population and the first real step to significantly improving racial diversity in the firefighting ranks. Yet those on the list awaiting appointment may find their hiring delayed as the judge looks to make whole those minority candidates who were victims of the flawed 1999 and 2002 exams.