A Guide to Connect Databricks and MongoDB Atlas using Python API

Overview

Integrating Databricks with MongoDB Atlas using Python API presents a potent solution for managing large-scale data and executing advanced analytics seamlessly. By following a structured guide, users can unite the collaborative analytics prowess of Databricks with MongoDB’s adaptable and scalable data storage capabilities, opening avenues for enriched data-driven insights and the development of machine learning applications.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Introduction

Databricks is a cloud-based analytical platform designed for processing and analyzing large volumes of big data. Developed by Microsoft, it leverages Apache Spark for computational tasks. With Databricks, users can seamlessly integrate their data, perform Extract, Load, Transform (ELT) processes, and implement machine learning workflows efficiently.

Operating on a parallel distributed system, Databricks automatically distributes workloads across multiple processors, ensuring high scalability and sharding. This distributed approach leads to decreased processing times and costs.

Key Characteristics of Databricks

Unified Platform: Databricks offers a unified platform that integrates data engineering, data science, and machine learning, streamlining the end-to-end analytics process.
Scalability: Built on Apache Spark, Databricks is scalable, enabling the efficient processing of large datasets across distributed clusters.
Collaboration: Databricks offers a shared workspace environment enabling teams to collaborate on notebooks, exchange insights, and cooperate on data initiatives.
Integrated Machine Learning: The platform includes MLlib, a scalable machine learning library, and supports MLflow for managing the end-to-end machine learning lifecycle.

MongoDB Atlas

MongoDB Atlas is a fully managed cloud database service tailored specifically for MongoDB users. It simplifies the deployment, administration, and scalability of MongoDB databases. With MongoDB Atlas, users can effortlessly set up, monitor, and expand their MongoDB databases in the cloud environment. This service offers a range of functionalities, including automated backups, comprehensive monitoring tools, and robust security measures.

Features of MongoDB Atlas

Automated Backups: MongoDB Atlas offers automated and continuous backups to ensure data durability and recovery options.
Scalability: Users can easily scale their MongoDB Atlas clusters vertically or horizontally to accommodate changing data needs.
Security Controls: Security features include encryption at rest, network isolation, and authentication mechanisms to protect data.
Monitoring and Alerts: MongoDB Atlas provides monitoring tools and configurable alerts to help users track the performance of their clusters.

Step-by-Step Guide

Step 1: Set up MongoDB Atlas

Visit the MongoDB Atlas website and sign in or create an account.
Create a new cluster, selecting your preferred cloud provider, region, and configuration options.
Allow the IP addresses of the Databricks clusters.
In the MongoDB Atlas dashboard, navigate to the “Network Access” section.
Add the IP addresses of your Databricks clusters to the whitelist to allow them to connect to MongoDB Atlas.

Step 2: Install Required Python Libraries

Run the following command in your Databricks notebook to install the pymongo library:

pythonCopy code
%pip install pymongo

1 2	pythonCopy code %pip install pymongo

Step 3: Connect Databricks Notebook to MongoDB Atlas

Code

from pymongo import MongoClient  

# Replace the following with your MongoDB Atlas connection string  
mongo_uri = "mongodb+srv://<username>:<password>@<cluster-url>/<database>?retryWrites=true&w=majority"  

# Connect to MongoDB Atlas  
client = MongoClient(mongo_uri)  

# Access your MongoDB database  
db = client.get_database('<database-name>')  

# Access your MongoDB collection  
collection = db.get_collection('<collection-name>')

from pymongo import MongoClient

# Replace the following with your MongoDB Atlas connection string

mongo_uri = "mongodb+srv://<username>:<password>@<cluster-url>/<database>?retryWrites=true&w=majority"

# Connect to MongoDB Atlas

client = MongoClient(mongo_uri)

# Access your MongoDB database

db = client.get_database('<database-name>')

# Access your MongoDB collection

collection = db.get_collection('<collection-name>')

Replace <username>, <password>, <cluster-url>, <database-name>, and <collection-name> with your MongoDB Atlas credentials and details.

Step 4: Perform CRUD Operations

Copy and paste the following code into your Databricks notebook to perform CRUD operations:

Code

# Insert data into MongoDB 
data_to_insert = {"key": "value"} 
collection.insert_one(data_to_insert) 

# Query data from MongoDB 
result = collection.find({"key": "value"}) 
for document in result: 
    print(document) 

# Update data in MongoDB 
collection.update_one({"key": "value"}, {"$set": {"key": "new-value"}}) 

# Delete data from MongoDB 
collection.delete_one({"key": "new-value"})

# Insert data into MongoDB

data_to_insert = {"key": "value"}

collection.insert_one(data_to_insert)

# Query data from MongoDB

result = collection.find({"key": "value"})

for document in result:

print(document)

# Update data in MongoDB

collection.update_one({"key": "value"}, {"$set": {"key": "new-value"}})

# Delete data from MongoDB

collection.delete_one({"key": "new-value"})

Step 5: Integrate with Databricks

In your Databricks workspace, create a new notebook.
Install the pymongo library using %pip install pymongo.
Copy and paste the code from Steps 3 and 4 into your Databricks notebook.

Conclusion

Integrating Databricks with MongoDB Atlas using Python API offers a powerful solution for easily managing and analyzing large-scale data. By seamlessly combining the collaborative analytics capabilities of Databricks with the scalable data storage features of MongoDB Atlas, organizations can unlock new opportunities for deriving insights and developing machine learning applications. This comprehensive guide has provided step-by-step instructions for establishing connectivity between Databricks and MongoDB Atlas, empowering users to leverage the strengths of both platforms effectively. This fusion of technologies makes data-driven decision-making more accessible, enabling organizations to extract maximum value from their data assets.

Drop a query if you have any questions regarding MongoDB and we will get back to you quickly.

Experience Effortless Cloud Migration with Our Expert Solutions

Stronger security
Accessible backup
Reduced expenses

Get Started

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, Microsoft Gold Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.

FAQs

1. What is Databricks, and how does it differ from Apache Spark?

ANS: – Databricks is a unified analytics platform built on Apache Spark, designed to simplify big data analytics. It provides an interactive workspace for data exploration, collaboration, and integration with various data sources. While Apache Spark is the underlying open-source distributed computing engine, Databricks adds a collaborative and user-friendly interface, making it easier for data engineers, data scientists, and analysts to work together seamlessly.

2. Can Databricks handle real-time data processing?

ANS: – Yes, Databricks supports real-time data processing by integrating with Apache Spark Streaming. You can build and deploy streaming applications in Databricks that process and analyze data in real-time. This allows for continuous insights and decision-making based on the most recent data.