Apple’s Innovative Leap with Open-Source Multimodal AI – Ferret

Introduction

In October 2023, in a surprising move, researchers from Columbia University and Apple Inc. quietly released Ferret, an open-source multimodal large language model. The introduction of Ferret on GitHub went unnoticed, lacking any official announcement or celebration. On October 30, the code for Ferret was quietly released, accompanied by Ferret-Bench. Checkpoint releases were subsequently introduced on December 14.

Unveiling both the code and weights, Apple took an unexpected turn, given its historical guardedness about tech releases. Ferret’s Multimodal Capabilities are widespread and are available in 7-billion and 13-billion parameter models. Its ability to analyze specific image regions and respond contextually to queries stands out. The smaller model is tailored for iOS devices, demonstrating Apple’s dedication to mobile efficiency.

Apple's Push for AI Integration

Apple’s recent research papers, focusing on deploying Large Language Models (LLMs) on phones, emphasize their commitment to integrating more AI components into devices. Ferret Bench, a benchmarking tool, assists researchers in evaluating its efficiency and flexibility across various use cases.

Ferret hints at Apple’s heightened commitment to transformer language models, signaling substantial enhancements for Siri and other language-related features. This model positions Apple as a frontrunner in multimodal AI capabilities, hinting at progress in AR/VR, camera technologies, and autonomous systems throughout Apple’s product range.

Technical Prowess and Performance

Powered by 8 Nvidia A100 GPUs, Ferret understands small image regions with minimal errors, showcasing Apple’s prowess in generative AI and multimodal capabilities. Trained on the GRIT dataset, it outshines its counterparts in referring and grounding tasks.

Ferret's Impact on Apple Devices

Ferret’s integration into Apple products promises revolutionary user experiences, including improved interactions with Siri, advanced visual searches, and enriched media understanding. Developers can leverage its capabilities for innovative applications across diverse domains.

Beyond Textual Comprehension

Ferret’s unique approach transcends textual comprehension, offering contextual responses by analyzing specific image regions. This sets a new standard in AI capabilities, providing deeper insights into visual content.

Ferret's Technical Aspects

Primarily a vision model, Ferret combines image and text understanding. Utilizing Clip Viit l14, it comprehends image content, identifies specific areas accurately, and understands complex shapes and details.

Benchmarking Against GPT 4 ROI

Benchmarked against GPT 4 ROI, Ferret outperformed in various aspects, showcasing advanced multimodal understanding and interaction capabilities.

Apple's Commitment to AI

Apple’s strategic acquisitions in the AI realm reflect its commitment to machine learning. These acquisitions enhance Apple’s AI capabilities, driving product and service innovation.

Glance at various company and their chatbots

OpenAI’s GPT-3 is leading the AI landscape for its understanding of natural language. Amazon Q is a new generative AI-powered assistant designed to solve problems, generate content, and gain insights. Microsoft Copilot can generate text conversationally, compose essays, create letters, summarize content, write code, and answer complex questions. Google’s Bard enhances search engine comprehension, and Anthropic pioneers AI research for ethical and robust language models. These diverse offerings showcase the industry’s commitment to advancing AI chatbots and large language models, catering to various user needs and applications.

Here is a blog on the Difference between ChatGPT and Google BARD.

Conclusion

The model uses a hybrid region representation and a spatial-aware visual sampler to enable fine-grained and open-vocabulary referring and grounding in the Multimodal Large Language Model. They also created the GRIT Dataset, a large-scale, hierarchical, robust ground-and-refer instruction tuning dataset containing approximately 1.1 million entries. The Ferret Model aims to refer and ground anything anywhere at any granularity, accepting any form of referring and grounding anything in response.

While Ferret proves to be a potent tool, it possesses specific constraints. As a relatively recent model, it may lack the robustness of more established counterparts. Similar to many other Machine Learning Language Models (MLLMs), Ferret carries the potential to generate responses that could be deemed harmful.

Additionally, it is crucial to highlight that Ferret is distributed under a non-commercial license. Apple envisions augmenting Ferret’s capabilities in the future to include the ability to generate segmentation masks alongside bounding boxes.

Drop a query if you have any questions regarding Ferret and we will get back to you quickly.

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, Microsoft Gold Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.