Measurements and Conclusions
As analysts, we rely upon research to form the basis of our work. If we’re conducting a 3D measurement exercise utilizing Single View Metrology, as operationalized by the programmers at Amped SRL, we’re relying not only upon the research of Antonio Criminisi and his team, but also the programmers who put his team’s work into their tool. We trust that all those involved in the process are acting in good faith. More often than not, analysts don’t dig around to check the research that forms the foundation of their work.
In our academic lives, we’ve conducted original research, we’ve supervised the research of others, we teach research methods, we’ve acted as an anonymous peer-reviewer, and we participate in and supervise an Institutional Review Boards (IRB). In our academic lives, as well as in our professional lives as an analysts, we use the model shown above to guide our work.
For those that are members of the IAI, and receive the Journal of Forensic Identification, you may have noticed that the latest edition features two letters to the editor that were submitted last summer. For those that don’t receive the JFI, you can follow along here but there’s no link that we can provide to share the letters as the JFI does not allow the general public to view it’s publications. Thus, we’re sharing a bit about our thought process in evaluating the particular article that deals with Forensic Photogrammetric Analysis (aka Photogrammetry).
The article that prompted the letter dealt with measuring 3D subjects depicted in a 2D medium (Meline, K. A., & Bruehs, W. E. (2018). A comparison of reverse projection and laser scanning photogrammetry. Journal of Forensic Identification, 68(2), 281-292), otherwise known as photogrammetry.
The first stop on the review was a foundational question, is the work experimental or non-experimental? There is a huge difference between the two. In experimental research an independent variable is manipulated in some way to find out how it will affect the dependent variable. For example, what happens to recorded frame rate in a particular DVR when all camera inputs are under motion or alarm mode? “Let’s put them all under load and see” tests the independent variables’ (camera inputs) potential influence on the dependent variable (recorded file) to find out if / how one affects the other. In non-experimental research there is no such manipulation, it’s largely observational. Experimental research can help to determine causes, as one is controlling the many factors involved. In general, non-experimental research can not help to determine causes. Experimental research is more time consuming to conduct, and thus more costly.
With this in mind, we read the paper looking for a description of the variables involved in the study – what was being studied and how potential influencing factors were being controlled. Finding none, we determined that the study was non-experimental – the researchers simply observed and reported.
The case study featured a single set of equipment and participants. The study did not examine the outputs from a range of randomly chosen DVRs paired with randomly chosen cameras. Nor did the study include a control group of participants. For the comparison of the methods studied, the study did not feature a range of laser scanners or a range of tools in which to create the reverse projection demonstratives. No discussion as to the specifics of the tool choices was given.
For the participants, those performing the measurement exam, no random assignment was used. Particularly troubling, the participants used in the study were co-workers of the researchers. Employees represent a vulnerable study population and problems can arise when these human subjects are not able to properly consent to participating in the research. Being involved in an institutional IRB, we would expect to see a statement about how the bias and conflicts would be mitigated in the research and that the researchers’ IRB had acknowledged and approved of the research design. Unfortunately, no such statement exists in the study. Given that the study was essentially a non-experimental test of human subjects, and not an experimental test of a particular technology or technique, an IRB’s approval is a must. One of the two letters that were submitted upbraided the editorial staff of the JFI for not enforcing it’s own rules as regards their requirement for an IRB approval statement for tests of human subjects.
Given the lack of control for bias and extraneous variables, the lack of randomness of participant selection, and the basic non-experimental approach of the work, we decided that this paper could not inform our work as analysts or our choice in employing a specific tool or technique.
Digging a bit deeper, we looked at the authors’ support for statements made – their references. We noticed that they chose to utilize some relatively obscure or largely unavailable sources. The chosen references would be unavailable to the average analyst without the analyst paying hundreds of dollars to check just one of the references. In our position, however, we have access to a world of research for free through our affiliations with various universities. So, we checked their references.
What we found, and thus reported to the Editor, was that many times the cited materials could not be applied to support the statements made in the research. In a few instances, the cited material actually refuted the authors’ assertions.
In the majority of the cited materials, the authors noted that their work couldn’t be used to inform a wider variety of research, that case-specific validity studies should be conducted by those engaged in the technique described, or that they were simply offering a “proof-of-concept” to the reader.
In evaluating this particular piece of research, we’re not looking to say – “don’t do this technique” or “this technique can’t be valid.” We want to know if we can use the study to inform our own work in performing photogrammetry. Unfortunately, due to the problems noted in my letters, we can’t.
If you’re interested in engaging in the creation of mixed-methods demonstratives for display in court, Occam’s Input-ACE has an overlay feature that allows one to mix the output from a laser scan into a project that contains CCTV footage. The overlay feature is comparable to a “reverse projection” – it’s a demonstrative used to illustrate and reinforce testimony. A “reverse projection” demonstrative is not, in and of itself, a measurement exercise / photogrammetry. Though it is possible to set up the demonstrative, then use SVM to measure within the space. If one wants to measure within the space in such a way as a general rule (not case specific), proper validity studies need to be conducted. At the time of the writing of this page (2019), no such validity studies exist for the calculation of measurements with such a mixed measures approach.
With all of this in mind, if one wants to form a conclusion as to a measure, one must move beyond simply drawing circles and arrows on a page. One must actually engage in a valid measurement experiment, using valid tools.
If you’re interested in a forensic photogrammetric analysis for your case, contact us today. If you’d like training / education on this topic, check out our calendar and sign up for the next class. Don’t see a date that works for you, suggest one. We can come to your location, our you can come to ours in Henderson, NV.