Featured Article

Glass rethinks the smartphone camera through an old-school cinema lens

Prepare to undergo anamorphosis

Comment

CG render of a phone with a prototype anamorphic lens.
Image Credits: Glass

Smartphone cameras have gotten quite good, but it’s getting harder and harder to improve them because we’ve pretty much reached the limit of what’s possible in the space of a cubic centimeter. Glass is a startup looking to fundamentally change how the camera works, using a much bigger sensor and an optical trick from the depths of filmmaking: anamorphic lenses.

It may not be obvious that cameras won’t get better, since we’ve seen such advances in recent generations of phones. But we’ve used up all the slack left in this line, as it were.

To improve the image, you need a bigger sensor, better lens or some kind of computational wizardry. Unfortunately, sensors can’t get much bigger because they’d need bigger lenses to match. And lenses can’t get bigger because there’s just no room for them in the phone body, even when you “fold” the camera. Meanwhile, computational photography is great, but there’s only so much it can do — stacking a few images to get better dynamic range or depth information is good, but you reach a point of diminishing returns pretty quickly.

“The limitations used to be about price, but now it’s size,” explained Glass co-founder and CEO Ziv Attar, who has worked in mobile imaging for over a decade, including at Apple. The other co-founder, Tom Bishop, also worked at Apple, the two of them working on creating Portrait Mode and likely chafing at the limitations of traditional camera design.

“Up to 5 years ago they just made the lens wider, then they started making the sensor bigger,” Attar said. “Then you throw algorithms at it to reduce noise, but even that is reaching its limits; pretty soon it will be pure hallucination [i.e. AI-generated imagery]. Night mode takes exposure stacking to extremes — it deals very nicely with the lack of photons, but if you zoom in it starts to look very weird and fake.”

“The phone screen kind of deceives us,” he continued. “If you let a regular person compare an iPhone 12 and 13, they won’t see the difference — but compared to a pro camera, anyone can tell. And if you can see the difference, there’s a lot of work to do.”

So what is that work, exactly? Attar has decided that of these various conundrums, the only one that makes sense to change is the lens. True, it can’t get any bigger — but only if you’re using a normal, symmetrical lens assembly. But why should we? They gave up on that constraint a century ago in cinema.

Anamorphic evolution

A CG image showing examples of anamorphic (top) and traditional symmetric lenses and the resulting internal image size..
A CG image showing examples of anamorphic (top) and traditional symmetric lenses and the resulting internal image size. Image Credits: Glass

Films weren’t always widescreen. Originally they were more likely to be approximately the shape of a 35mm film frame, for obvious reasons. If you matted out the top and bottom, you could project a widescreen image, which people liked — but you were basically just zooming in on a part of the film, which you paid for in detail. But a technique first tested in the ’20s soon solved the problem.

Anamorphic lenses squeeze a wide field of view from the sides so it fits in the film frame, and when projected using an anamorphic projector, the process was reversed — the image is stretched back out to the desired aspect ratio. There are a few interesting optical effects introduced but… if I describe them you’ll never be able to un-see them in content, so I’ll forbear.

The lens system proposed by Glass isn’t quite the same, but it uses similar principles and unusually shaped lenses. It started from the fundamental idea of how to add a larger sensor. Simply making a larger square would necessitate a larger lens, which we can’t do — but what if you made the sensor longer, as in a rectangle? Well, you’d need a longer, rectangular lens too. The anamorphic technique means you can capture and project a larger but distorted image, then convert it to the right aspect ratio in the image processor. (The process isn’t exactly analogous to the film technique but it uses the same principles.)

How much larger an image are you able to capture? Well, an iPhone 13’s main camera has a sensor about 7×5 millimeters, so 35 square mm total. Glass’s prototype uses a sensor that’s about 24×8 mm: about 192 square mm, 5-6 times larger, with a commensurate increase to megapixels. Here’s a little chart for casual reference:

Image Credits: Devin Coldewey / TechCrunch

Considering the fanfare that generally accompanies increasing a phone’s sensor size by 15 or 20 percent, that’s an enormous leap.

But Attar explained that the way they measure it, it’s even more. If you were to expand the image to the correct aspect ratio, it would actually be twice as tall: 24x16mm, just shy of the APS-C standard in DSLRs but well above the Micro Four Thirds and 1″ sensors also common (and highly performant) in mirrorless cameras. That leads to the company’s claim of having 11 times the “imaging area” of an iPhone. The evaluation of these metrics is a non-trivial process I’m not equipped to do, but truthfully either one would be a game-changing upgrade for a phone.

Bigger, brighter and a bit weirder

There are benefits and drawbacks to this process. The most important one is an immense increase in light gathering and resolving power. More light means better exposures in general and better shots in challenging conditions — no need for a fancy machine learning powered multi-exposure night mode if you can just… see things. And there is far, far more detail in images compared with those from ordinary smartphones.

Images from an iPhone 12 Max (left) and Glass prototype. Image Credits: Glass

Note that the limited example above is just that — it’s hard to do apples-to-apples comparisons when the focal lengths, image processing and output resolution are so different (not to mention my cropping and re-encoding), but at the very least you can see that a great deal of detail is added even in this non-optimal presentation. The full-size original images are available here: iPhone, Glass.

An example of a very low-light exposure — iPhone left, Glass right. Image Credits: Glass

Because of the larger sensor and the nature of the glass, you also get natural bokeh, or background blur. Portrait mode is of course a favorite among smartphone users, but even the best methods of simulating bokeh are far from perfect. The same effect Apple painstakingly simulated lenses to achieve occurs naturally on the Glass prototype, just as it would on a larger digital camera. And there’s no chance of the kind of weird mistake you see in the AI-segmented images, which often clip out hair and other details, or fail to achieve the depth effect in subtler ways.

Example image showing portrait mode on an iPhone (left) and the unprocessed Glass shot, which lacks the smoothing and artifacts of the manipulated one. Image Credits: Glass

While there would be no optical zoom, Attar pointed out that zooming in by cropping (i.e. digital zoom) on a Glass system would let you zoom in more than most optical zooms out there, and you’d still have more light and pixels than the competition. I’m not normally one to let “digital zoom is fine” claims live, but in this case the sheer size of the lens and sensor more than make up for it.

These benefits, though briefly stated, are more than considerable. The improvement to light and detail puts it way out in front of the best cameras out there. (And while the smallest details may escape your notice on a small screen, a bad exposure is noticeable at any size.)

Drawbacks are mainly to do with the complexities of operating a camera that’s totally optically different from a traditional one. The mechanisms for autofocus are different (anamorphic focus is notoriously complex) and there are plenty of distortions and aberrations that need to be corrected for — symmetrical lenses at this size also have distortion and degradation, but of a different type.

“[Distortions] are all constrained during design such that we know in advance that we can correct for them,” said Attar. “It’s an iterative process but we did kick start development of a custom dedicated software tool to co-optimize lens parameters and neural network variables.” In other words they didn’t design anything they couldn’t correct for.

One effect I find disorientating but perhaps others will decide is trivial is the shape of the bokeh. Normally out of focus highlights blur out into little translucent discs, but in the Glass system they resolve into a gradient of ovals and chubby crescents.

Example image showing background blur behind a plant stem.
Image Credits: Glass

To my neurotic eye that just isn’t right. It’s… unnatural. But I also can’t not notice vignetted bokeh due to french flags in film and TV (don’t look it it up — this too is everywhere and you can’t unsee it). And anyway films shot in anamorphic show similar bokeh distortion, so it’s actually quite common, just not in still images and smartphone shots.

I assumed there would be drawbacks due to the need to stretch the image digitally — that sort of thing if done poorly can lead to moiré and other unwanted artifacts. But Attar said it’s remarkably straightforward to train a model to do it so that no one can tell the difference except pixel peepers: “We trained networks to apply 1-D super-resolution based on information from the other axes. After we apply our algorithm it looks like it came from a full APS-C sensor, in field of view and resolution.”

That will all have to be verified by reviewers and camera experts when there’s a production version, but the theory seems sound and the early results are more than promising.

Right now the company has moved on from standalone prototypes to a third-generation phone factor device that shows how the tech will fit into pretty much any chassis on the market. There’s nothing exotic about it other than the optical qualities, Attar said, so although it won’t be as cheap to manufacture as today’s off-the-shelf camera and image processing units, it can be made just as easily. As he noted, price is hardly an option any more, and if one company can make a huge leap in camera quality they can capture a large chunk of the market.

“We have to convince a phone maker to basically ditch the old technology,” said Attar. “We’re seeing nice feedback. The only challenge is doing it in a reasonable time. I’m not saying there’s no risk. But a lot of us had good jobs at big companies — we didn’t leave our fancy salaries at Apple to work on some BS thing. We had a plan from the beginning.”

Even if an agreement was struck now with a big mobile manufacturer, it would take a year and half or two years to get to market. “But we have to start somewhere,” he concluded.

Glass has raised $2.2 million in seed funding, led by LDV Capital and a collection of angel investors. Of course that’s not meant to cover the cost of manufacturing, but now that the company is leaving the lab it will need operating cash to commercialize even should a major manufacturer make a commitment. Greg Gilley, formerly Apple’s VP of cameras and photos, and MIT Media lab’s Ramesh Raskar joined as advisors, rounding out a team investors are likely to have a lot of confidence in.

If the Glass approach catches on, expect to hear about other companies claiming to have invented it in a little less than two years.

More TechCrunch

Poshmark, the social commerce site that lets people buy and sell new and used items to each other, launched a paid marketing tool on Thursday, giving sellers the ability to…

Poshmark’s ‘Promoted Closet’ tool lets sellers boost all their listings at once

Google is launching a Gemini add-on for educational institutes through Google Workspace.

Google adds Gemini to its Education suite

More money for the generative AI boom: Y Combinator-backed developer infrastructure startup Recall.ai announced Thursday it’s raised a $10 million Series A funding round, bringing its total raised to over $12M.…

YC-backed Recall.ai gets $10M Series A to help companies use virtual meeting data

Engineers Adam Keating and Jeremy Andrews were tired of using spreadsheets and screenshots to collab with teammates — so they launched a startup, Colab, to build a better way. The…

Colab’s collaborative tools for engineers line up $21M in new funding

Reddit announced on Wednesday that it is reintroducing its awards system after shutting down the program last year. The company said that most of the mechanisms related to awards will…

Reddit reintroduces its awards system

Sigma Computing, a startup building a range of data analytics and business intelligence tools, has raised $200 million in a fresh VC round.

Sigma is building a suite of collaborative data analytics tools

European Union enforcers of the bloc’s online governance regime, the Digital Services Act (DSA), said Thursday they’re closely monitoring disinformation campaigns on the Elon Musk-owned social network X (formerly Twitter)…

EU ‘closely’ monitoring X in wake of Fico shooting as DSA disinfo probe rumbles on

Wind is the largest source of renewable energy in the U.S., according to the U.S. Energy Information Administration, but wind farms come with an environmental cost as wind turbines can…

Spoor uses AI to save birds from wind turbines

The key to taking on legacy players in the financial technology industry may be to go where they have not gone before. That’s what Chicago-based Aeropay is doing. The provider…

Cannabis and gaming payments startup Aeropay is now offering an alternative to Mastercard and Visa

Facebook and Instagram are under formal investigation in the European Union over child protection concerns, the Commission announced Thursday. The proceedings follow a raft of requests for information to parent…

EU opens child safety probes of Facebook and Instagram, citing addictive design concerns

Bedrock Materials is developing a new type of sodium-ion battery, which promises to be dramatically cheaper than lithium-ion.

Forget EVs: Why Bedrock Materials is targeting gas-powered cars for its first sodium-ion batteries

Private equity giant Thoma Bravo has announced that its security information and event management (SIEM) company LogRhythm will be merging with Exabeam, a rival cybersecurity company backed by the likes…

Thoma Bravo’s LogRhythm merges with Exabeam in more cybersecurity consolidation

Consumer protection groups around the European Union have filed coordinated complaints against Temu, accusing the Chinese-owned ultra low-cost e-commerce platform of a raft of breaches related to the bloc’s Digital…

Temu accused of breaching EU’s DSA in bundle of consumer complaints

Here are quick hits of the biggest news from the keynote as they are announced.

Google I/O 2024: Here’s everything Google just announced

The AI industry moves faster than the rest of the technology sector, which means it outpaces the federal government by several orders of magnitude.

Senate study proposes ‘at least’ $32B yearly for AI programs

The FBI along with a coalition of international law enforcement agencies seized the notorious cybercrime forum BreachForums on Wednesday.  For years, BreachForums has been a popular English-language forum for hackers…

FBI seizes hacking forum BreachForums — again

The announcement signifies a significant shake-up in the streaming giant’s advertising approach.

Netflix to take on Google and Amazon by building its own ad server

It’s tough to say that a $100 billion business finds itself at a critical juncture, but that’s the case with Amazon Web Services, the cloud arm of Amazon, and the…

Matt Garman taking over as CEO with AWS at crossroads

Back in February, Google paused its AI-powered chatbot Gemini’s ability to generate images of people after users complained of historical inaccuracies. Told to depict “a Roman legion,” for example, Gemini would show…

Google still hasn’t fixed Gemini’s biased image generator

A feature Google demoed at its I/O confab yesterday, using its generative AI technology to scan voice calls in real time for conversational patterns associated with financial scams, has sent…

Google’s call-scanning AI could dial up censorship by default, privacy experts warn

Google’s going all in on AI — and it wants you to know it. During the company’s keynote at its I/O developer conference on Tuesday, Google mentioned “AI” more than…

The top AI announcements from Google I/O

Uber is taking a shuttle product it developed for commuters in India and Egypt and converting it for an American audience. The ride-hail and delivery giant announced Wednesday at its…

Uber has a new way to solve the concert traffic problem

Google is preparing to launch a new system to help address the problem of malware on Android. Its new live threat detection service leverages Google Play Protect’s on-device AI to…

Google takes aim at Android malware with an AI-powered live threat detection service

Users will be able to access the AR content by first searching for a location in Google Maps.

Google Maps is getting geospatial AR content later this year

The heat pump startup unveiled its first products and revealed details about performance, pricing and availability.

Quilt heat pump sports sleek design from veterans of Apple, Tesla and Nest

The space is available from the launcher and can be locked as a second layer of authentication.

Google’s new Private Space feature is like Incognito Mode for Android

Gemini, the company’s family of generative AI models, will enhance the smart TV operating system so it can generate descriptions for movies and TV shows.

Google TV to launch AI-generated movie descriptions

When triggered, the AI-powered feature will automatically lock the device down.

Android’s new Theft Detection Lock helps deter smartphone snatch and grabs

The company said it is increasing the on-device capability of its Google Play Protect system to detect fraudulent apps trying to breach sensitive permissions.

Google adds live threat detection and screen-sharing protection to Android

This latest release, one of many announcements from the Google I/O 2024 developer conference, focuses on improved battery life and other performance improvements, like more efficient workout tracking.

Wear OS 5 hits developer preview, offering better battery life