Sony partners with Microsoft to achieve its AI powered image sensor ambitions
Capable of performing local processing, a key to advancing autonomous cars
Only days after Sony unveiled the unique and new IMX500 and IMX501, a pair of image sensors tacked on top of AI powered chips which provide local- and cloud-computing processing, the company has provided a lot of additional details on their ambitions.
While I don’t want to seem as if I’m taking a victory lap, it’s a good feeling to have your opinions and viewpoints later validated by the company. The original story about the Intelligent Vision Sensors as Sony brands them is as follows:
As I wrote, combining an image sensor with an AI chip isn’t all that new and in fact, is the sole reason behind the computational photography revolution:
On paper, the idea isn’t all that new, and companies like Apple and Google have been doing it with their smartphones for years. […]
The upside of democratizing such a sensor is that it allows others to provide AI and Machine Learning without needing the expertise and backing that Apple and Google enjoy. It’s unlikely that those companies would use or need to use such sensors as they’ll build their own. But for smaller players coming into the market, this could be a game-changer that gives them access to a subset of tech that may have otherwise been unattainable.
Vlad Savov and Takashi Mochizuki on Bloomberg¹:
Apple Inc. has already proven the efficacy of combining AI and imaging to create more secure systems with its Face ID biometric authentication, powered by the iPhone’s custom-designed Neural Engine processor. Huawei Technologies Co. and Alphabet Inc.’s Google also have dedicated AI silicon in their smartphones to assist with image processing. These on-device chips represent what’s known as edge computing: handling complex AI and machine-learning tasks at the so-called edge of the network instead of sending data back and forth to servers.
If Sony doesn’t wish to compete with Apple, then who is their target buyer? Corroborating my sentiment is Hideki Somemiya, senior general manager of Sony’s System Solutions group:
We are aware many companies are developing AI chips and it’s not like we try to make our AI chip better than others. Our focus is on how we can distribute AI computing across the system, taking cost and efficiency into consideration. Edge computing is a trend, and in that respect, ours is the edge of the edge.
Now it’s worth noting that in some ways, Sony was never going to ‘compete’ head-to-head with Apple in the traditional sense because their Bionic and ARM chipsets are created with the express purpose of being used within their own product lines like iPhone, iPad, Apple TV, and maybe one day Mac. Apple isn’t interested in being a supplier to others. However, if Sony provided something better than what they could create on their own, like image sensors, which Apple does source from them, then they very well could have tapped Sony for their expertise. The same applies to Google.
With Sony relatively new to the world of AI and Machine Learning, a key component of their new stacked chipsets, they’ve turned to an old competitor of theirs to help bridge the gap — Microsoft.
Sony Corp. and Microsoft Corp. have partnered to embed artificial intelligence capabilities into the Japanese company’s latest imaging chip, a big boost for a camera product the electronics giant describes as a world-first for commercial customers.
The new module’s big advantage is that it has its own processor and memory built in, which allows it to analyze video using AI tech like Microsoft’s Azure, but in a self-contained system that’s faster, simpler and more secure to operate than existing methods.
In hindsight, Sony tapping Microsoft for Azure isn’t all that surprising and, in fact, is the second time they’ve done so in the last year.
So how does Sony hope to leverage its image sensor technology and Azure’s AI capabilities? From yours truly:
The upside of democratizing such a sensor is that it allows others to provide AI and Machine Learning without needing the expertise and backing that Apple and Google enjoy. It’s unlikely that those companies would use or need to use such sensors as they’ll build their own. But for smaller players coming into the market, this could be a game-changer that gives them access to a subset of tech that may have otherwise been unattainable.
And that’s exactly who Sony is targeting with these new chips. Vlad and Takashi:
The two companies are appealing to retail and logistics businesses with potential uses like optimizing warehouse and factory automation, quantifying the flow of customers through stores and making cars smarter about their drivers and environment.
At CES 2020, Sony showcased just that with Vision-S, a prototype autonomous car that houses an array of the company’s image sensors which better allow it to see the world in real-time.
Of course, an all-seeing chip that’s capable of processing so much data and identifying it accurately poses enormous privacy concerns as I wrote:
In the right hands, these sensors can change how we shop and interact with the world as we’ve always dreamt of in sci-fi flicks. But as the saying goes, with great power comes great responsibility, and it’s just as easy for such a system to be abused — one that erodes our privacy even more and further analysis our every move in public.
Luckily, Sony is aware of these concerns, and one of the foundational pieces of their sensors is the ability to process data and transmit information while respecting our privacy. Vlad:
Instead of generating actual images, Sony’s AI chip can analyze the video it sees and provide just metadata about what’s in front of it — saying instead of showing what’s in its frame of vision. Because no data is sent to remote servers, opportunities for hackers to intercept sensitive images or video are dramatically reduced, which should help allay privacy fears.
The ability for these chipsets to process data locally means we may finally begin to see meaningful advancements in autonomous driving that go beyond the highway. Like humans, an autonomous car has to be capable of driving whenever, wherever, in any condition, and cannot be reliant on cloud computing to analyze, process, and respond to the world around it. While connectivity will offer additional benefits, your car should be able to do all this anywhere without a network connection, and therefore cloud computing. Otherwise, without a proper connection, your autonomous car will suddenly be exponentially more dangerous and less reliable.
The IMX500 and IMX501 chips tackle that with their local processing capabilities.
The AI-powered Sony sensor is capable of recording high-resolution video and simultaneously conducting its AI analysis at up to 30 frames each second. That rapid, up-to-the-microsecond responsiveness makes it potentially suitable for in-car use such as detecting when a driver is falling asleep, Sony’s Somemiya said. Without the need for a “cloud brain” as some existing systems have, Sony’s AI sensor could hasten the adoption of smart-car technology.
What’s unclear about the partnership is the extent that Microsoft, and, in turn, Azure is being leveraged — is local processing that sends data to a connected computer (in the case of autonomous driving, the onboard computer provided by the carmaker) being handled entirely by Sony — and only when the chips need to offload data to the cloud is Azure the ‘brain’ that crunches the data? Or, in both cases, even for local data analysis, is Azure a key component of it all?
It’s been quite clear to me for some time (and I’ve written extensively about it) that Sony’s future is being built upon the shoulders of PlayStation and their image sensors. As we move towards a more connected world where not just our phones, but our cars, retail stores, drones, and factories need to be able to see, process, and understand the world around them, it creates a near infinitely high demand for sensors, a market of which Sony already controls 51%. By tapping Microsoft as a partner to help them make their chips smarter, Sony is now positioned to further dominate and extend their lead and increase their clout in the market.
Do you think it’s wise for Sony to team up with Microsoft, or should they bring every aspect of their operations and ambitions in-house?
¹ John Gruber routinely calls out Bloomberg for their bullshit and has written this amazing footer to go along with them.
Bloomberg, of course, is the publication that published “The Big Hack” in October — a sensational story alleging that data centers of Apple, Amazon, and dozens of other companies were compromised by China’s intelligence services. The story presented no confirmable evidence at all, was vehemently denied by all companies involved, has not been confirmed by a single other publication (despite much effort to do so), and has been largely discredited by one of Bloomberg’s own sources. By all appearances “The Big Hack” was complete bullshit. Yet Bloomberg has issued no correction or retraction, and seemingly hopes we’ll all just forget about it. I say we do not just forget about it. Bloomberg’s institutional credibility is severely damaged, and everything they publish should be treated with skepticism until they retract the story or provide evidence that it was true.
Thanks for being an SRN reader. If you liked this post and want to help me cover Sony even more with reviews, videos, and podcasts, please clap (you can 👏 more than once!), share on your favorite social sites, and subscribe to the SRN newsletter.
