Recognition and Modeling of Planar Mechanical Linkages from Images Using Symbolic and Behavioral Cues

From windshield wipers to selfie sticks, many of the machines and consumer products we use rely on mechanical linkages to accomplish functional goals. Even though we observe the dynamic behavior of these mechanisms on a daily basis, during the design and analysis of such systems, the visual content is largely static. Students may be forced to grapple with abstract depictions found in textbooks, and engineers may use hand-drawn sketches to brainstorm design ideas. Current software tools are ill-suited for fast kinematic visualization of mechanical designs and may require expertise that hinders novice users. With that in mind, the goal of this work is to create a computational method capable of quickly generating accurate kinematic models from images of planar mechanical linkages. Despite remarkable progress in recent years for computer vision tasks such as object recognition, scene understanding, and image segmentation, the problem of identifying a collection of connected parts with unknown structure in an image is a challenging one. Our framework leverages supervised learning methods for localizing mechanical parts (e.g. joints, rigid bodies) with the optimizing power of a multiobjective evolutionary algorithm for predicting feasible topologies. We systematically evaluate each stage of our framework and introduce a novel metric called the user effort ratio to compare the overall performance of different algorithms and assess the benefit of automatic recognition over manual model construction. The final outcome is a standalone software application that takes a raw image as input and produces a kinematic simulation of the pictured mechanism with minimal user interaction.