“THE BODY DOESN’T WALK BY ITSELF.”

ABOUT

In the context of COVID-19, how might we provide a better remote interaction experience for physical therapists and patients sheltering at home? How might we create meaningful interactions, both on and off the screen? How might we leverage gameplay to motivate daily engagement? How might we receive and transmit physical sensations between people? These and many other questions formed the basis of our design process and research.

OUR SOLUTION

RE°FLEX IS A MULTIMODAL INTERACTION SYSTEM DESIGNED TO ENHANCE LIVE REMOTE COLLABORATION AND COMMUNICATION BETWEEN PHYSICAL THERAPISTS AND THEIR PATIENTS 

The goals for this project included:

• Exploration: investigate human needs through research, interviews, and evaluation of works by other practitioners.

• Synthesis: distill research insights to identify design opportunities and produce a set of criteria.

• Iterative design: create representations and artifacts of potential solutions, prototyping toward a refined concept.

This project was collaboration by Yueru Deng, Matt Geiger, Lulin Shan, and Catherine Yochum. in Interaction Design Studio, taught by Kyuha Shim and Dina El-Zanfaly at the School of Design, Carnegie Mellon University.

1.   Identifying the problem: Moments of Mismatch

Interviewing stakeholders revealed several challenges, as well as viable opportunities for improvement

• Video calls: this mode of communication provides some aspects of face-to-face communication, but lacks subtle queues such as eye contact, directed speech, and a sense of space. While this medium works fairly well for one-on-one communications, it begins to lose clarity and effectiveness when used for larger groups.

• Body language: webcam and video teleconference systems were designed to facilitate remote meetings, where most body language happens from a seated position. The optics in webcams are usually mounted on or integrated above computer displays. This design is intended to work for subjects who are sitting within arm’s reach of their device. Physical activity requires greater distance.

• Touch: sports training, martial arts, physical therapy—these kinds of activities engage our tactile senses. The lack of touch and physical interaction over video conferencing systems is a major obstacle for physical therapy in a remote context. We explored haptics and mixed reality technologies as possible interventions.

2. THE CURRENT STATE: Existing Technologies

Many remote interaction technologies are designed to address a specific scenario (or a narrow set of scenarios). Considering the scenarios described by our interviewees, we evaluated existing technologies to identify opportunities for improved remote interactions.

To provide meaningful interactions between the physical therapist and patients, we considered a range of platforms and technologies. From our research and interviews, we found a need for better representation of touch beyond vibrating motors.

3. Ideation: User Needs

We considered using independent servo motors connected by high strength nylon filament (i.e., fishing line) to recreate the feeling of pulling. This mechanism would be capable of wirelessly transmitting gestures for pose and form corrections. While not intended for a final end-user product, this approach helped us to consider other ways to include directionality to remote tactile feedback.

To grapple with the complexity of this large problem space, we decided to narrow our focus onto knee injuries and recovery. In our second pass of prototyping, we envisioned an intelligent knee-brace that could capture real-time movement data, and provide targeted haptic feedback to patients. This approach involved the use of embedded LRA (linear Resonant Actuators) devices and accelerometers. The placement of devices above and below the knee would allow for directional and three-dimensional haptic feedback to patients.

As we continued to explore this problem space, including additional interviews, we found that seamlessness is an important consideration. Bulky devices allow for including more robust features and integration, but at the cost of simplicity and wearability. This is an extraordinary drawback for patients who may desire an all-day wearable device, or something that can fit underneath their clothing.

For considerations of a 3rd-generation device, we focused on simplifying the placement and application of devices as well as scalability. The initial question for this pivot in form was: how big is too big? Our second question: how small, and where on the body? While our consideration of a knee brace offered some flexibility, the use of a more modular system presents a much wider range of options. Physical prototyping and evaluation helped to narrow the scope. This evaluation suggested that the dimensions for the adhesive patches needs some revision to operate well on this region of the body. The ultimate goal is to design nodes that are exactly as small as they need to be, but no smaller. The lower limit is critical.

Generally speaking, the point of prototyping is to show and tell. Show: what you have in mind, and tell: what that would be like. This process revealed a narrow range of acceptable sizes for nodes (devices combining motion tracking and haptic feedback). These constraints also provided some much needed priorities. Wearable devices are limited by battery capacity, weight, and size. What can be achieved within these limitations matters.

Another consideration: based on feedback we received from physical therapists, it became clear that we needed to design a system with components that differ between the patient and therapist experience. These stakeholders have different needs, and we envisioned a bigger technology investment (into things like a Mixed Reality headset) would be beneficial to the physical therapists. On the patient side, the Reflex Nodes must provide a layer of guidance, adding a felt sense to augment the visual and verbal instructions from the physical therapist.

The nodes are designed to address two needs: motion tracking and haptic feedback. Their shape, placement, and components were continuously revised throughout the design process.