Why Robots Still Cannot Reliably Pick Things Up

A robotic arm in a modern logistics centre can move with sub-millimetre precision along a pre-programmed path. It can operate for years without fatigue, maintaining a level of repeatability that surpasses human capability. Yet, if a single plastic bottle in a bin is slightly dented, or if a silicon spatula is resting at an unfamiliar angle, the machine often fails. The arm might crush the object, drop it, or miss it entirely. This inability to reliably grasp and manipulate everyday items remains one of the most stubborn bottlenecks in robotics.

The difficulty does not stem from a lack of computing power or motor control. It arises from the fundamental complexity of physical contact. When a human picks up a pen, the brain integrates visual data with a constant stream of tactile feedback, adjusting grip force and finger placement in real time. We perceive the texture, the weight distribution, and the slight slide of the object against our skin. Robots, by contrast, largely operate in a state of sensory deprivation regarding the physical interaction between their grippers and the world.

Most successful robotic deployments avoid these problems by removing uncertainty. In car manufacturing, every part is held in a specific jig at a known coordinate. The robot does not need to sense the part; it merely needs to move to a set of coordinates. Outside these controlled environments, in places like hospitals, kitchens, or recycling plants, the geometry and material properties of objects are unpredictable. The physics governing these interactions are non-linear and difficult to model, leading to a gap between what a robot can see and what it can effectively handle.

The gap between seeing and touching

Computer vision has made remarkable progress over the last decade, particularly through the use of deep convolutional neural networks. A robot can now identify a mug or a screwdriver within a cluttered scene with high statistical confidence. However, identifying an object is not the same as understanding how to interact with it. A visual system provides a three-dimensional map of an object’s surface, but it cannot see the object’s centre of mass, its friction coefficient, or its internal structural integrity.

This distinction creates a significant data gap. A vision system might perceive a cardboard box as a rigid cube, but it cannot know if the box is empty or filled with heavy lead weights until contact is made. If the robot applies a standard grip to a heavy box, the item will slip. If it applies the same force to a thin plastic cup, the cup may collapse. The transition from a visual representation to a successful physical manipulation requires an estimation of hidden properties that light alone cannot convey.

Furthermore, occlusion presents a constant hurdle. As a robotic hand approaches an object, the hand itself eventually blocks the cameras’ view of the contact point. At the most critical moment of the task—the millisecond before and during the grip—the robot is often effectively blind. It must rely on a mental model of the scene that may no longer be accurate if the object shifts or rolls during the approach. Without sophisticated tactile feedback to replace the lost visual data, the robot is operating on an outdated map.

Geometry and the mathematics of the grip

The mathematical framework for grasping is often described in terms of force closure and form closure. Form closure occurs when the geometry of the gripper entirely surrounds an object, preventing any movement regardless of the force applied. A hand cupping a ball is a rudimentary example. Force closure is more common and more complex; it occurs when the combination of contact forces and friction can resist any external force trying to move the object.

Calculating the ideal points for force closure is a significant computational challenge. For a simple rigid shape like a brick, the math is straightforward. For a soft or irregularly shaped object, such as a bag of flour or a bunch of keys, the number of potential contact points is infinite. The robot must solve for a configuration where the fingers apply enough pressure to generate friction, but not so much that they damage the object or cause it to squirt out of the grip.

The challenge lies in the fact that physical contact is not a single event but a continuous negotiation between surfaces.

Current algorithms often attempt to simplify this by identifying a few key points on an object’s surface. They calculate a grasp quality metric based on the proximity of these points to the estimated centre of mass. However, these models frequently assume that objects are rigid and that the points of contact will not deform. In reality, almost all domestic objects deform under pressure. When a gripper squeezes a sponge or a piece of fruit, the geometry of the contact area changes, shifting the very points the algorithm was relying on to maintain stability.

The physics of frictional uncertainty

Friction is the primary force that allows us to hold things, yet it is one of the most difficult physical properties to predict accurately. At a microscopic level, friction is the result of complex interactions between surface asperities and molecular bonds. In robotics, this is usually simplified into a single number: the coefficient of friction. This simplification is rarely sufficient for reliable manipulation in the real world.

The friction between a robotic fingertip and an object can change based on humidity, the presence of dust, or the wear on the gripper’s surface material. If an object is slightly damp, the friction may drop precipitously, causing a planned grasp to fail. Moreover, the distribution of friction across a contact patch is rarely uniform. When a robot turns a doorknob or twists a cap, it relies on torsional friction, which is even harder to model than simple sliding friction.

Simulating these effects is computationally expensive. Most robotic training happens in digital environments, where physics engines use approximations to keep the simulation running quickly. These approximations often fail to capture the subtle slip-and-stick phenomena that occur just before an object falls. This discrepancy is known as the sim-to-real gap. A robot may learn to pick up a virtual bottle a million times in a simulator, but because the simulator’s friction model is too perfect, the robot fails when it encounters a real bottle with a slightly oily surface.

Limitations of current tactile sensor hardware

To overcome the limits of vision and the uncertainty of friction, robots need a sense of touch comparable to human skin. However, the hardware for tactile sensing is decades behind the hardware for vision. While high-resolution cameras are cheap and ubiquitous, high-resolution tactile sensors are largely confined to research laboratories. Most industrial grippers are equipped with only basic force sensors that measure the total load on the motor, rather than the specific distribution of pressure across the finger.

Developing a robust tactile sensor involves a difficult trade-off between sensitivity and durability. A sensor must be soft enough to conform to objects and detect light pressures, yet tough enough to withstand thousands of cycles of grinding against rough metal or sharp edges. Many experimental sensors use arrays of tiny electrodes or optical fibers embedded in a soft polymer. While these can provide detailed images of the contact area, they are often fragile and difficult to repair when the polymer skin eventually tears.

Another significant hurdle is the wiring and data integration. A human hand has thousands of nerve endings providing a massive, parallel stream of data to the brain. Replicating this in a robotic digit requires managing hundreds of individual sensor signals within a very small volume. Currently, the most advanced tactile sensors, such as those that use internal cameras to track the deformation of a rubber skin, produce large amounts of data that are difficult to process in real time. The robot ends up with a high-definition picture of its own finger deforming, but it still lacks the sophisticated software architecture to turn that image into a decisive physical correction.

The computational cost of real-time deformation

The physical reality of picking up an object involves constant, minute adjustments. When a human finger presses against a surface, the flesh deforms, increasing the surface area and redistributing the pressure. This deformation is not a single state but a dynamic process. For a robot to replicate this, it must calculate the stresses across a non-rigid surface in milliseconds. The computational burden is substantial because the physics of soft materials, known as continuum mechanics, is mathematically more complex than the physics of rigid bodies.

Most robotic controllers rely on rigid-body dynamics, which assumes that parts do not change shape. When a robot attempts to handle a soft or slippery object, this assumption fails. To account for deformation, the system must solve partial differential equations that describe how the sensor skin stretches and compresses. Even with modern graphics processing units, doing this at the speeds required for reactive balancing—such as catching a slipping glass—remains difficult. If the processing lag exceeds a few dozen milliseconds, the robot’s corrective movement will likely arrive too late, often overcompensating and causing the object to be dropped or crushed.

Furthermore, the data generated by high-resolution tactile sensors is often redundant. A camera-based sensor may produce a video stream of several megapixels, yet only a few dozen pixels actually represent the critical point of slippage. Distinguishing the signal from the noise requires significant local processing power. Engineers are currently exploring neuromorphic chips, which process data in a manner similar to biological neurons, to handle these tactile inputs. These chips only respond to changes in the signal, such as a sudden shift in texture, which could potentially reduce the power and time required to detect a slip.

Why simulation fails to capture reality

To train the software that governs these movements, researchers often turn to computer simulations. It is faster and safer to have a virtual hand drop a virtual bottle ten million times than to do so in a physical laboratory. However, these simulations frequently suffer from what is known as the reality gap. The models used to simulate friction, stiction, and material deformation are approximations that do not fully capture the chaotic nature of the physical world.

The primary difficulty lies in the fact that friction is not a single constant value but a transient phenomenon influenced by microscopic surface variations.

In a simulator, two surfaces are often treated as perfectly smooth or having a uniform coefficient of friction. In reality, a plastic bottle may have a thin film of condensation, a slight oily residue from a previous handler, or a microscopic seam from the manufacturing process. These variables change how the object behaves when touched. When a robot trained exclusively in simulation is placed in a real kitchen, it often fails because it has not learned to account for these subtle, unmodelled physical forces.

There is also the problem of contact dynamics. When a robot makes contact with an object, the forces spike instantaneously. Simulators struggle to calculate these impulse forces accurately without crashing or producing physically impossible results, such as the hand passing through the object. To prevent this, developers often soften the physics in the simulation, making the world appear more viscous or "rubbery" than it actually is. This helps the software run smoothly but results in a robot that is too timid or imprecise when handling firm, real-world materials.

Data scarcity in robotic manipulation

A deeper issue involves the sheer lack of data available for robotic learning. Large language models succeed because they are trained on trillions of words found on the internet. There is no equivalent dataset for physical touch. While there are vast libraries of video showing humans handling objects, these videos do not record the forces involved. A robot can see how a hand moves, but it cannot see the pressure exerted by the fingertips or the friction between the skin and the surface.

Collecting this data is a slow, manual process. Unlike digital data, physical data requires hardware that is subject to wear and tear. If a laboratory runs a fleet of robots for twenty-four hours a day to collect grasping data, the robots will eventually break, require recalibration, or need their tactile skins replaced. This makes the cost of data acquisition in robotics orders of magnitude higher than in other fields of artificial intelligence.

Currently, the largest datasets for robotic manipulation contain roughly a few million trials. While this sounds significant, it is small when considering the infinite variety of object shapes, weights, and textures. A robot that learns to pick up a red cube in a lab may not immediately understand how to pick up a translucent plastic cup or a crumpled bag of crisps. The lack of a "foundation model" for physics means that most robots are still specialists, trained for specific tasks in controlled environments rather than being general-purpose tools.

Alternative approaches through reinforcement learning

To overcome the limitations of manual programming, researchers are increasingly using reinforcement learning. In this approach, the robot is not told exactly how to move. Instead, it is given a goal—such as "hold the object upright"—and a reward signal when it succeeds. Through millions of trials, the robot discovers its own strategies for stabilization. This often leads to emergent behaviours that look surprisingly human, such as using a thumb to brace an object against the palm.

Reinforcement learning is particularly effective at finding ways to exploit the passive physics of the hand. For instance, a robot might learn that it does not need to calculate the exact pressure required if it simply leans the object against its own frame for support. However, this method is black-box in nature. It is often difficult for engineers to understand why the robot chose a specific movement, which makes it hard to guarantee safety or reliability in high-stakes environments like hospitals.

Recent experiments have combined reinforcement learning with "domain randomization." By intentionally varying the physics in the simulator—changing the gravity, the friction, and the weight of objects randomly—researchers force the robot to develop a robust control strategy that can handle uncertainty. If a robot can learn to pick up a block when the simulated friction is changing every second, it is more likely to handle the unpredictable surfaces of the real world. This approach has shown promise, but it still requires a final phase of "fine-tuning" on real hardware, which remains the bottleneck of the process.

The path toward reliable general-purpose handling

The ability for a robot to walk into an unfamiliar room and pick up an arbitrary object remains an unsolved problem. What has been established is that rigid, programmed movements are insufficient for the complexity of the physical world. There is a consensus among researchers that successful manipulation will require a combination of high-resolution tactile sensing and machine learning models that can generalize from limited data.

Several developments could change the current trajectory. The first would be the invention of a tactile skin that is as durable as human skin and as easy to manufacture as a silicon chip. This would lower the barrier to collecting large-scale tactile data. The second would be a breakthrough in "cross-modal" learning, where robots learn about physics by watching videos of humans and inferring the hidden forces through advanced physical modelling.

For now, robots remain most effective in environments where the objects are known and the lighting is consistent. The transition from the factory floor to the home requires a level of physical intelligence that is not yet achievable. While robots can now perform impressive feats in controlled demonstrations, their reliability in the chaotic, unscripted conditions of everyday life is still far below the threshold required for widespread adoption. The gap between seeing an object and securely holding it is not just a matter of better motors, but a fundamental challenge in how machines perceive and react to the resistance of the physical world.