Featured Article

What is a liquid neural network, really?

We spoke with MIT CSAIL head Daniela Rus about the emerging technology and implications for robotics

Comment

Image Credits: Getty Images

The initial research papers date back to 2018, but for most, the notion of liquid networks (or liquid neural networks) is a new one. It was “Liquid Time-constant Networks,” published at the tail end of 2020, that put the work on other researchers’ radar. In the intervening time, the paper’s authors have presented the work to a wider audience through a series of lectures.

Ramin Hasani’s TEDx talk at MIT is one of the best examples. Hasani is the Principal AI and Machine Learning Scientist at the Vanguard Group and a Research Affiliate at CSAIL MIT, and served as the paper’s lead author.

“These are neural networks that can stay adaptable, even after training,” Hasani says in the video, which appeared online in January. When you train these neural networks, they can still adapt themselves based on the incoming inputs that they receive.”

The “liquid” bit is a reference to the flexibility/adaptability. That’s a big piece of this. Another big difference is size. “Everyone talks about scaling up their network,” Hasani notes. “We want to scale down, to have fewer but richer nodes.” MIT says, for example, that a team was able to drive a car through a combination of a perception module and liquid neural networks comprised of a mere 19 nodes, down from “noisier” networks that can, say, have 100,000.

“A differential equation describes each node of that system,” the school explained last year. “With the closed-form solution, if you replace it inside this network, it would give you the exact behavior, as it’s a good approximation of the actual dynamics of the system. They can thus solve the problem with an even lower number of neurons, which means it would be faster and less computationally expensive.”

The concept first crossed my radar by way of its potential applications in the robotics world. In fact, robotics make a small cameo in that paper when discussing potential real-world use. “Accordingly,” it notes, “a natural application domain would be the control of robots in continuous-time observation and action spaces where causal structures such as LTCs [Liquid Time-Constant Networks] can help improve reasoning.”


Want the top robotics news in your inbox each week? Sign up for Actuator here.


One of the benefits of these systems is that they can be run with less computing power. That means — potentially — using something as simple as a Raspberry Pi to execute complex reasoning, rather than offloading the task to external hardware via the cloud. It’s easy to see how this is an intriguing solution.

Another potential benefit is the black box question. This refers to the fact that — for complex neural networks — researchers don’t entirely understand how the individual neurons combine to create their final output. One is reminded of that famous Arthur C. Clarke quote, “Any sufficiently advanced technology is indistinguishable from magic.” This is an oversimplification, of course, but the point remains: Not all magic is good magic.

The adage “garbage in, garbage out” comes into play here, as well. We all intuitively understand that bad data causes bad outputs. This is one of the places where biases become a factor, impacting the ultimate result. More transparency would play an important role in identifying causality. Opacity is a problem — particularly in those instances where inferred causality could be — quite literally — a matter of life or death.

Paper co-author and head of MIT CSAIL, Daniela Rus, points me in the direction of a fatal Tesla crash in 2016, where the imaging system failed to distinguish an oncoming tractor trailer from the bright sky that backdropped it. Tesla wrote at the time:

Neither Autopilot nor the driver noticed the white side of the tractor trailer against a brightly lit sky, so the brake was not applied. The high ride height of the trailer combined with its positioning across the road and the extremely rare circumstances of the impact caused the Model S to pass under the trailer, with the bottom of the trailer impacting the windshield of the Model S.

Hasani says the fluid systems are “more interpretable,” due, in part, to their smaller size. “Just changing the representation of a neuron,” he adds, “you can really explore some degrees of complexity you couldn’t explore otherwise.

As for the downsides, these systems require “time series” data, unlike other neural networks. That is to say that they don’t currently extract the information they need from static images, instead requiring a dataset that involves sequential data like video.

“The real world is all about sequences,” says Hasani. “Even our perception — you’re not perceiving images, you’re perceiving sequences of images,” he says. “So, time series data actually create our reality.”

I sat down with Rus to discuss the nature of these networks, and help my creative writing-major brain understand about what the emerging technology could mean for robotics, moving forward.

Image Credits: MIT CSAIL

TechCrunch: I get the sense that one of the motivating factors [in the creation of liquid neural networks] are hardware constraints with mobile robotic systems and their inability to run extremely complex computations and rely on these very complex neural networks.

Daniela Rus: That is part of it, but most importantly, the reason we started thinking about liquid networks has to do with some of the limitations of today’s AI systems, which prevent them from being very effective for safety, critical systems and robotics. Most of the robotics applications are safety critical.

I assume most of the people who have been working on neural networks would like to do more with less if that were an option.

Yes, but it’s not just about doing more with less. There are several ways in which liquid networks can help robotic applications. From a computational point of view, the liquid networkers are much smaller. They are compact, and they can run on a Raspberry Pi and EDGE devices. They don’t need the cloud. They can run on the kinds of hardware platforms that we have in robotics.

We get other benefits. They’re related to whether you can understand how the machine learning system makes decisions — or not. You can understand whether the decision is going to meet some safety constraints or not. With a huge machine learning model, it is actually impossible to know how that model makes decisions. The machine learning models that are based on deep networks are really huge and opaque.

And unpredictable.

They are opaque and unpredictable. This presents a challenge, as users are unable to comprehend how decisions are made by these systems. This is really a problem if you want to know if your robot will move correctly — whether the robot will respond correctly to a perceptual input. The first fatal accident for Tesla was due to a machine learning imperfection. It was due to a perception error. The car was riding on the highway, there was a white truck that was crossing and the perceptual system believed that the white truck was a cloud. It was a mistake. It was not the kind of mistake that a human would make. It’s the kind of mistake that a machine would make. In safety critical systems, we cannot afford to have those kinds of mistakes.

Having a more compact system, where we can show that the system doesn’t hallucinate, where we can show that the system is causal and where we can maybe extract a decision tree that presents in understandable human form how the system reaches a conclusion, is an advantage. These are properties that we have for liquid networks. Another thing we can do with liquid networks is we can take a model and wrap it in something we call “BarrierNet.” That means adding another layer to the machine learning engine that incorporates the constraints that you would have in a control barrier function formulation for control theoretic solutions for robots. What we get from BarrierNet is a guarantee that the system will be stable and fall within some bounds of safety.

You’re talking about intentionally adding more constraints.

Yes, and they are the kind of constraints that we see in control theoretic, model predictive control solutions for robots. There is a body of work that is developing something called “control barrier functions.” It’s this idea that you have a function that forces the output of a system within a safe region. You could take a machine learning system, wrap it with BarrierNet, and that filters and ensures that the output will fall within the safety limits of the system. For example, BarrierNet can be used to make sure your robot will not exert a force or a torque outside its range of capability. If you have a robot that aims to apply force that is bigger than what it can do, then the robot will break itself. There are examples of what we can do with BarrieNet and Liquid Networks. Liquid networks are excitig because they address issues that we have with some of today’s systems.

They use less data and less training time. They’re smaller, and they use much less inference time. They are provably causal. They have action that is much more focused on the task than other machine learning approaches. They make decisions by somehow understanding aspects of the task, rather than the context of the task. For some applications, the context is very important, so having a solution that relies on context is important. But for many robotic applications, the context is not what matters. What matters is how you actually do the task. If you’ve seen my video, it shows how a deep network would keep a car on the road by looking at bushes and trees on the side of the road. Whereas the liquid networks have a different way of deciding what to do. In the context of driving, that means looking at the road’s horizon and the side of the road. This is, in some sense, a benefit, because it allows us to capture more about the task.

It’s avoiding distraction. It’s avoiding being focused on every aspect of the environment.

Exactly. It’s much more focused.

Do these systems require different kinds and amounts of training data than more traditional networks?

In our experiments, we have trained all the systems using the same data, because we want to compare them with respect to performance after training them in the same way. There are ways to reduce the data that is needed to train a system, and we don’t know yet whether these data reduction techniques can be applied broadly — in other words, they can benefit all models. Or if they are particularly useful for a model that is formulated in the dynamical systems formulation that we have for liquid networks.

I want to add that liquid neural networks require time series data. A liquid network solution does not work on datasets consisting of static data items. We cannot use ImageNet with liquid neural networks right now. We have some ideas, but as of now, one really needs an application with time series to use liquid networks.

The methods for reducing the amount of information that we feed into our models are somehow related to this notion that there’s a lot of repetition. So not all the data items give you effective new information. There is data that brings and widens information diversity, and then there’s data that’s more of the same. For our solutions, we have found that if we can curate the datasets so that they have diversity and less repetition, then that is the fine way to move forward.

Ramin [Hasani] gave a TEDx talk at MIT. One of the things he said, which makes a lot of sense on the face of it is, “bad data means bad performance.”

Garbage in, garbage out.

Is the risk of bad data heightened or mitigated when you’re dealing with a smaller number of neurons?

I don’t think it’s any different for our model than any other model. We have fewer parameters. The computations in our models are more powerful. There’s task equivalence.

If the network is more transparent, it’s easier to determine where the problems are cropping up.

Of course. And if the network is smaller, it’s easier from the point of view of a human. If the model is small and only has a few neurons, then you can visualize the performance. You can also automatically extract a decision tree for how the model makes decisions.

When you talk about causal relationships, it sounds like one of the problems with these massive neural networks is that they can connect things almost arbitrarily. They’re not necessarily determining causal relationships in each case. They’re taking extra data and making connections that aren’t always there.

The fact is that we really don’t understand. What you say sounds right, intuitively, but no one actually understands how the huge models make decisions. In a convolutional solution to a perception problem, we think that we understand that the earlier layers essentially identify patterns at small scale, and the later layers identify patterns at higher levels of abstraction. If you look at a large language model and you ask, ‘how does it reach decisions?’ You will hear that we don’t really know. We want to understand. We want to study these large models in greater detail and right now this is difficult because the models are behind walls.

Do you see generative AI itself having a major role in robotics and robotics learning, going forward?

Yes, of course. Generative AI is extraordinarily powerful. It really put AI in everyone’s pockets. That’s extraordinary. We didn’t really have a technology like this before now. Language has become a kind of operating system, a programming language. You can specify what you want to do in natural language. You don’t need to be a rock-star computer scientist to synthesize a program. But you also have to take this with a grain of salt, because the results that are focused on code generation show that it’s mostly the boiler plate code that can be generated automatically by large language models. Still, you can have a co-pilot that can help you code, catch mistakes, basically the co-pilot helps programmers be more productive.

So, how does it help in robotics? We have been looking at using generative AI to solve complex robotics problems that are difficult to solve with our existing methods […] With generative AI you can get much faster solutions and much more fluid and human-like solutions for control than we get with model predictive solutions. I think that’s very powerful. The robots of the future will move less mechanistically. They’ll be more fluid, more humanlike in their motions. We’ve also used generative AI for design. Here we need more than pattern generation for robots. The machines have to make sense in the context of physics and the physical world.

More TechCrunch

Zoox, Amazon’s self-driving unit, is bringing its autonomous vehicles to more cities.  The self-driving technology company announced Wednesday plans to begin testing in Austin and Miami this summer. The two…

Zoox to test self-driving cars in Austin and Miami 

Called Stable Audio Open, the generative model takes a text description and outputs a recording up to 47 seconds in length.

Stability AI releases a sound generator

It’s not just instant-delivery startups that are struggling. Oda, the Norway-based online supermarket delivery startup, has confirmed layoffs of 150 jobs as it drastically scales back its expansion ambitions to…

SoftBank-backed grocery startup Oda lays off 150, resets focus on Norway and Sweden

Newsletter platform Substack is introducing the ability for writers to send videos to their subscribers via Chat, its direct messaging feature, the company announced on Wednesday. The rollout of video…

Substack brings video to its Chat feature

Hiya, folks, and welcome to TechCrunch’s inaugural AI newsletter. It’s truly a thrill to type those words — this one’s been long in the making, and we’re excited to finally…

This Week in AI: Ex-OpenAI staff call for safety and transparency

Ms. Rachel isn’t a household name, but if you spend a lot of time with toddlers, she might as well be a rockstar. She’s like Steve from Blues Clues for…

Cameo fumbles on Ms. Rachel fundraiser as fans receive credits instead of videos  

Cartwheel helps animators go from zero to basic movement, so creating a scene or character with elementary motions like taking a step, swatting a fly or sitting down is easier.

Cartwheel generates 3D animations from scratch to power up creators

The new tool, which is set to arrive in Wix’s app builder tool this week, guides users through a chatbot-like interface to understand the goals, intent and aesthetic of their…

Wix’s new tool taps AI to generate smartphone apps

ClickUp Knowledge Management combines a new wiki-like editor and with a new AI system that can also bring in data from Google Drive, Dropbox, Confluence, Figma and other sources.

ClickUp wants to take on Notion and Confluence with its new AI-based Knowledge Base

New York City, home to over 60,000 gig delivery workers, has been cracking down on cheap, uncertified e-bikes that have resulted in battery fires across the city.  Some e-bike providers…

Whizz wants to own the delivery e-bike subscription space, starting with NYC

This is the last major step before Starliner can be certified as an operational crew system, and the first Starliner mission is expected to launch in 2025. 

Boeing’s Starliner astronaut capsule is en route to the ISS 

TechCrunch Disrupt 2024 in San Francisco is the must-attend event for startup founders aiming to make their mark in the tech world. This year, founders have three exciting ways to…

Three ways founders can shine at TechCrunch Disrupt 2024

Google’s newest startup program, announced on Wednesday, aims to bring AI technology to the public sector. The newly launched “Google for Startups AI Academy: American Infrastructure” will offer participants hands-on…

Google’s new startup program focuses on bringing AI to public infrastructure

eBay’s newest AI feature allows sellers to replace image backgrounds with AI-generated backdrops. The tool is now available for iOS users in the U.S., U.K., and Germany. It’ll gradually roll…

eBay debuts AI-powered background tool to enhance product images

If you’re anything like me, you’ve tried every to-do list app and productivity system, only to find yourself giving up sooner than later because sooner than later, managing your productivity…

Hoop uses AI to automatically manage your to-do list

Asana is using its work graph to train LLMs with the goal of creating AI assistants that work alongside human employees in company workflows.

Asana introduces ‘AI teammates’ designed to work alongside human employees

Taloflow, an early stage startup changing the way companies evaluate and select software, has raised $1.3M in a seed round.

Taloflow puts AI to work on software vendor selection to reduce cost and save time

The startup is hoping its durable filters can make metals refining and battery recycling more efficient, too.

SiTration uses silicon wafers to reclaim critical minerals from mining waste

Spun out of Bosch, Dive wants to change how manufacturers use computer simulations by both using modern mathematical approaches and cloud computing.

Dive goes cloud-native for its computational fluid dynamics simulation service

The tension between incumbents and fintechs has existed for decades. But every once in a while, the two groups decide to put their competition aside and work together. In an…

When foes become friends: Capital One partners with fintech giants Stripe, Adyen to prevent fraud

After growing 500% year-over-year in the past year, Understory is now launching a product focused on the renewable energy sector.

Insurance provider Understory gets into renewable energy following $15M Series A

Ashkenazi will start her new role at Google’s parent company on July 31, after 23 years at Eli Lilly.

Alphabet brings on Eli Lilly’s Anat Ashkenazi as CFO

Tobiko aims to reimagine how teams work with data by offering a dbt-compatible data transformation platform.

With $21.8M in funding, Tobiko aims to build a modern data platform

In 1816, French physician René Laennec invented an instrument that allowed doctors to listen to the heart and lungs. That device — a stethoscope — eventually evolved from a simple…

Eko Health scores $41M to detect heart and lung disease earlier and more accurately

The number of satellites on low Earth orbit is poised to explode over the coming years as more mega-constellations come online. This will create new opportunities for bad actors to…

DARPA and Slingshot build system to detect ‘wolf in sheep’s clothing’ adversary satellites

SAP sees WalkMe’s focus on automating contextual, in-app support as bringing value to its own enterprise customers.

SAP to acquire digital adoption platform WalkMe for $1.5B

The National Democratic Alliance (NDA) has emerged victorious in India’s 2024 general election, but with a smaller majority compared to 2019. According to post-election analysis by Goldman Sachs, JPMorgan, CLSA,…

Modi-led coalition’s election win signals policy continuity in India — and spending cuts

Featured Article

A comprehensive list of 2024 tech layoffs

The tech layoff wave is still going strong in 2024. Following significant workforce reductions in 2022 and 2023, this year has already seen 60,000 job cuts across 254 companies, according to independent layoffs tracker Layoffs.fyi. Companies like Tesla, Amazon, Google, TikTok, Snap and Microsoft have conducted sizable layoffs in the…

22 hours ago
A comprehensive list of 2024 tech layoffs

Featured Article

What to expect from WWDC 2024: iOS 18, macOS 15 and so much AI

Apple is hoping to make WWDC 2024 memorable as it finally spells out its generative AI plans.

23 hours ago
What to expect from WWDC 2024: iOS 18, macOS 15 and so much AI

We just announced the breakout session winners last week. Now meet the roundtable sessions that really “rounded” out the competition for this year’s Disrupt 2024 audience choice program. With five…

The votes are in: Meet the Disrupt 2024 audience choice roundtable winners