Sergey Levine - Building LLMs for the Physical World - [Invest Like the Best, EP.465]

“What if we just add more data labeled with the semantic command? So basically, just take whatever the robot experienced and just label it with some semantic commands, but don't add any more low-level actions. And that actually helps.”

“LLMs make certain kinds of representations very convenient. They make it very convenient to basically turn text into other text. But that's not necessarily the best representation for what an embodied system needs to do. Sometimes it needs to think about things more spatially. Sometimes semantically, sometimes other representations.”

“getting to a particular level of usefulness so that robots can be deployed so they can do useful tasks so they can start collecting data from open world settings at scale. Because that's such a sudden event, getting past the activation energy, I think there is a lot of uncertainty about the timing of that.”