Optimizing Messaging Costs with AWS IoT Core’s Basic Ingest

Overview

In today’s digital era, where the Internet of Things (IoT) reigns supreme, managing data transfer efficiently and cost-effectively is paramount. Enter AWS IoT Core’s Basic Ingest, a game-changing feature designed to simplify data ingestion processes while slashing operational costs. By bypassing traditional messaging methods, Basic Ingest offers a streamlined approach to transmitting IoT data to various AWS services. In this blog, we’ll explore how Basic Ingest is transforming the landscape of IoT data management, enabling organizations to leverage the full potential of AWS IoT Core while saving money.

Introduction

With so many IoT devices around and tons of data being generated, saving on messaging costs has become a big deal for businesses everywhere.

This new way of doing things not only makes operations smoother but also lets businesses expand their IoT projects without breaking the bank.

Message broker

AWS IoT’s message broker allows devices and clients to publish and subscribe to messages, which are then distributed to subscribed devices and clients. Additionally, the message broker forwards messages to the rule’s engine for further processing.

Basic Ingest

Basic Ingest, a new feature offered by AWS IoT Core, enhances the process of ingesting high-volume data workloads by bypassing the pub/sub–Message Broker from the ingestion path. Doing so enables customers to send device data to other AWS services in a more cost-effective way while still leveraging the security and data processing capabilities of AWS IoT Core.

Basic Ingest is now supported in multiple AWS regions worldwide, including US GovCloud, US East, US West, Asia Pacific, and Europe. This enables customers to use the feature in various locations, depending on their data processing requirements.

Basic Ingestion topic

Basic Ingest, a feature of AWS IoT Core, allows for the secure transmission of device data to AWS services supported by AWS IoT rule actions.

Basic Ingest uses topic names that start with “$aws/rules/rule_name” for the first three levels of messages sent from devices or applications. This enables users to invoke an existing rule by simply adding the Basic Ingest prefix to the message topic they would typically use. For example, if a rule named “EV_data” is invoked by messages with topics like “EV/device1/Temperature” (“sql”: “SELECT * FROM ‘EV/#'”), users can invoke the same rule with Basic Ingest by sending a message with the topic “$aws/rules/EV_data/EV/device1/Temperature”.

Steps need to be taken care of while using Basic Ingest

1. The prefix of a Basic Ingest topic, which is $aws/rules/rule_name, cannot be utilized by the topic(Decimal)
2. To ensure that your device or application can use Basic Ingest, you need to verify that it has the appropriate permissions to publish on $aws/rules/* within the policy. Alternatively, you can grant permission for specific rules using $aws/rules/rule_name/* in the policy. If these conditions are not met, you can still use your current AWS IoT Core connections for your devices and applications.
3. It is worth noting that there is no variation in execution or error handling between rules triggered by Basic Ingest and those triggered via message broker subscriptions once the rules engine receives the message.
4. Your devices or rules can’t subscribe to Basic Ingest reserved topics.
5. In the case where you create a rule that is only triggered by Basic Ingest, the FROM clause within the sql field of the rule definition is not mandatory. However, suppose the rule is also invoked by other messages that need to be transmitted through the message broker (such as messages that must be disseminated to multiple subscribers). In that case, the FROM clause remains a requirement. You can refer to the AWS IoT SQL reference for further details on this topic (AWS IoT SQL reference).
6. When it comes to the Basic Ingest topic, which is represented by $aws/rules/rule_name, the first three levels are not included in the 8-segment length restriction or the 256-character limit for a topic. However, all other constraints, as specified in the AWS IoT Limits documentation, still apply (AWS IoT Limits).
7. If a message is received with a Basic Ingest topic that references an inactive or non-existent rule, an error log is generated in an Amazon CloudWatch log to aid in troubleshooting. Additional details can be found in the Rules engine log entries documentation. Furthermore, a metric called RuleNotFound is presented, and it’s possible to establish alarms for this metric using the instructions available in the Rule Metrics documentation.
8. It is possible to publish messages with a QoS of 1 on Basic Ingest topics. After the message is delivered to the rule’s engine, a PUBACK is sent to confirm successful delivery. However, it is important to note that receiving a PUBACK does not necessarily indicate that your rule actions were executed without errors. To handle errors that may occur during action execution, you can configure an error action. Further details regarding error handling can be found in the Error Handling (error action) documentation.
9. Suppose you require a publish/subscribe broker to disseminate messages to numerous subscribers (e.g., delivering messages to other devices and the rules engine). In that case, it is recommended to continue using the AWS IoT message broker to manage message distribution. However, it’s crucial to ensure that messages are published on topics that are not Basic Ingest topics.

Conclusion

Basic ingestion offers a cost-effective and efficient alternative to traditional IoT data ingestion methods by bypassing the pub/sub–Message Broker. By streamlining the data transmission process and preserving the security and data processing capabilities of AWS IoT Core, basic ingestion empowers organizations to build scalable and robust IoT solutions that meet their business needs effectively.

Drop a query if you have any questions regarding AWS IoT Core and we will get back to you quickly.

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, Microsoft Gold Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.