Amazon DocumentDB zero-ETL integration with Amazon OpenSearch Service is now obtainable

In the present day, we’re saying the overall availability of Amazon DocumentDB (with MongoDB compatibility) zero-ETL integration with Amazon OpenSearch Service.

Amazon DocumentDB supplies native textual content search and vector search capabilities. With Amazon OpenSearch Service, you may carry out superior search analytics, resembling fuzzy search, synonym search, cross-collection search, and multilingual search, on Amazon DocumentDB knowledge.

Zero-ETL integration simplifies your structure for superior search analytics. It frees you from performing undifferentiated heavy lifting duties and the prices related to constructing and managing knowledge pipeline structure and knowledge synchronization between the 2 companies.

On this put up, we present you configure zero-ETL integration of Amazon DocumentDB with OpenSearch Service utilizing Amazon OpenSearch Ingestion. It includes performing a full load of Amazon DocumentDB knowledge and constantly streaming the newest knowledge to Amazon OpenSearch Service utilizing change streams. For different ingestion strategies, see documentation.

Answer overview

At a excessive stage, this answer includes the next steps:

1. Allow change streams on the Amazon DocumentDB collections.
2. Create the OpenSearch Ingestion pipeline.
3. Load pattern knowledge on the Amazon DocumentDB cluster.
4. Confirm the info in OpenSearch Service.

Stipulations

To implement this answer, you want the next stipulations:

Zero-ETL will carry out an preliminary full load of your assortment by doing a set scan on the first occasion of your Amazon DocumentDB cluster, which can take a number of minutes to finish relying on the scale of the info, and chances are you’ll discover elevated useful resource consumption in your cluster.

Allow change streams on the Amazon DocumentDB collections

Amazon DocumentDB change stream occasions comprise a time-ordered sequence of knowledge modifications attributable to inserts, updates, and deletes in your knowledge. We use these change stream occasions to transmit knowledge modifications from the Amazon DocumentDB cluster to the OpenSearch Service area.

Change streams are disabled by default; you may allow them on the particular person assortment stage, database stage, or cluster stage. To allow change streams in your collections, full the next steps:

1. Hook up with Amazon DocumentDB utilizing mongo shell.
2. Allow change streams in your assortment with the next code. For this put up, we use the Amazon DocumentDB database stock and assortment product:

   db.adminCommand(modifyChangeStreams: 1,
       database: "stock",
       assortment: "product", 
       allow: true);

In case you have a couple of assortment for which you wish to stream knowledge into OpenSearch Service, allow change streams for every assortment. If you wish to allow it on the database or cluster stage, see Enabling Change Streams.

It’s really helpful to allow change streams for under the required collections.

Create an OpenSearch Ingestion pipeline

OpenSearch Ingestion is a completely managed knowledge collector that delivers real-time log and hint knowledge to OpenSearch Service domains. OpenSearch Ingestion is powered by the open supply knowledge collector Knowledge Prepper. Knowledge Prepper is a part of the open supply OpenSearch venture.

With OpenSearch Ingestion, you may filter, enrich, rework, and ship your knowledge for downstream evaluation and visualization. OpenSearch Ingestion is serverless, so that you don’t want to fret about scaling your infrastructure, working your ingestion fleet, and patching or updating the software program.

For a complete overview of OpenSearch Ingestion, go to Amazon OpenSearch Ingestion, and for extra details about the Knowledge Prepper open supply venture, go to Knowledge Prepper.

To create an OpenSearch Ingestion pipeline, full the next steps:

1. On the OpenSearch Service console, select Pipelines within the navigation pane.
2. Select Create pipeline.
3. For Pipeline identify, enter a reputation (for instance, zeroetl-docdb-to-opensearch).
   
4. Arrange pipeline capability for compute sources to mechanically scale your pipeline primarily based on the present ingestion workload.
5. Enter the minimal and most Ingestion OpenSearch Compute Items (OCUs). On this instance, we use the default pipeline capability settings of minimal 1 Ingestion OCU and most 4 Ingestion OCUs.

Every OCU is a mix of roughly 8 GB of reminiscence and a couple of vCPUs that may deal with an estimated 8 GiB per hour. OpenSearch Ingestion helps as much as 96 OCUs, and it mechanically scales up and down primarily based in your ingest workload demand.

6. Select the configuration blueprint and underneath Use case within the navigation pane, select ZeroETL.
7. Choose Zero-ETL with DocumentDB to construct the pipeline configuration.

This pipeline is a mix of a supply half from the Amazon DocumentDB settings and a sink half for OpenSearch Service.

You could set a number of AWS Identification and Entry Administration (IAM) roles (sts_role_arn) with the required permissions to learn knowledge from the Amazon DocumentDB database and assortment and write to an OpenSearch Service area. This function is then assumed by OpenSearch Ingestion pipelines to verify the fitting safety posture is at all times maintained when shifting the info from supply to vacation spot. To be taught extra, see Organising roles and customers in Amazon OpenSearch Ingestion.

You want one OpenSearch Ingestion pipeline per Amazon DocumentDB assortment.

model: "2"
documentdb-pipeline:
  supply:
    documentdb:
      acknowledgments: true
      host: "<<docdb-2024-01-03-20-31-17.cluster-abcdef.us-east-1.docdb.amazonaws.com>>"
      port: 27017
      authentication:
        username: $aws_secrets:secret:username
        password: $aws_secrets:secret:password
      aws:
        sts_role_arn: "<<arn:aws:iam::123456789012:function/Instance-Function>>"
      
      s3_bucket: "<<bucket-name>>"
      s3_region: "<<bucket-region>>" 
      # non-obligatory s3_prefix for Opensearch ingestion to write down the information
      # s3_prefix: "<<path_prefix>>"
      collections:
        # assortment format: <databaseName>.<collectionName>
        - assortment: "<<databaseName.collectionName>>"
          export: true
          stream: true
  sink:
    - opensearch:
        # REQUIRED: Present an AWS OpenSearch endpoint
        hosts: [ "<<https://search-mydomain-1a2a3a4a5a6a7a8a9a0a9a8a7a.us-east-1.es.amazonaws.com>>" ]
        index: "<<index_name>>"
        index_type: customized
        document_id: "$getMetadata("primary_key")"
        motion: "$getMetadata("opensearch_action")"
        # DocumentDB file creation or occasion timestamp
        document_version: "$getMetadata("document_version")"
        document_version_type: "exterior"
        aws:
          # REQUIRED: Present a Function ARN with entry to the area. This function ought to have a belief relationship with osis-pipelines.amazonaws.com
          sts_role_arn: "<<arn:aws:iam::123456789012:function/Instance-Function>>"
          # Present the area of the area.
          area: "<<us-east-1>>"
          # Allow the 'serverless' flag if the sink is an Amazon OpenSearch Serverless assortment
          # serverless: true
          # serverless_options:
            # Specify a reputation right here to create or replace community coverage for the serverless assortment
            # network_policy_name: "network-policy-name"
          
extension:
  aws:
    secrets and techniques:
      secret:
        # Secret identify or secret ARN
        secret_id: "<<my-docdb-secret>>"
        area: "<<us-east-1>>"
        sts_role_arn: "<<arn:aws:iam::123456789012:function/Instance-Function>>"
        refresh_interval: PT1H 

Present the next parameters from the blueprint:

• Amazon DocumentDB endpoint – Present your Amazon DocumentDB cluster endpoint.
• Amazon DocumentDB assortment – Present your Amazon DocumentDB database identify and assortment identify within the format dbname.assortment throughout the col­­­­­lections part. For instance, stock.product.
• s3_bucket – Present your S3 bucket identify together with the AWS Area and S3 prefix. This shall be used quickly to carry the info from Amazon DocumentDB for knowledge synchronization.
• OpenSearch hosts – Present the OpenSearch Service area endpoint for the host and supply the popular index identify to retailer the info.
• secret_id – Present the ARN for the key for the Amazon DocumentDB cluster together with its Area.
• sts_role_arn – Present the ARN for the IAM function that has permissions for the Amazon Doc DB cluster, S3 bucket, and OpenSearch Service area.

To be taught extra, see Creating Amazon OpenSearch Ingestion pipelines.

5. After getting into all of the required values, validate the pipeline configuration for any errors.
   
6. When designing a manufacturing workload, deploy your pipeline inside a VPC. Select your VPC, subnets, and safety teams. Additionally choose Connect to VPC and select the corresponding VPC CIDR vary.

The safety group inbound rule ought to have entry to the Amazon DocumentDB port. For extra data, consult with Securing Amazon OpenSearch Ingestion pipelines inside a VPC.

Load pattern knowledge on the Amazon DocumentDB cluster

Full the next steps to load the pattern knowledge:

1. Hook up with your Amazon DocumentDB cluster.
2. Insert some paperwork into the gathering product within the stock database by working the next instructions. For creating and updating paperwork on Amazon DocumentDB, consult with Working with Paperwork.

   use stock;
   
    db.product.insertMany([
      
         "Item":"Ultra GelPen",
         "Colors":[
            "Violet"
         ],
         "Stock":
            "OnHand":100,
            "MinOnHand":35
         ,
         "UnitPrice":0.99
      ,
      
         "Merchandise":"Poster Paint",
         "Colours":[
            "Red",
            "Green",
            "Blue",
            "Black",
            "White"
         ],
         "Stock":
            "OnHand":47,
            "MinOnHand":50
         
      ,
      
         "Merchandise":"Spray Paint",
         "Colours":[
            "Black",
            "Red",
            "Green",
            "Blue"
         ],
         "Stock":
            "OnHand":47,
            "MinOnHand":50,
            "OrderQnty":36
         
      
   ])

Confirm the info in OpenSearch Service

You need to use the OpenSearch Dashboards dev console to seek for the synchronized gadgets inside just a few seconds. For extra data, see Creating and trying to find paperwork in Amazon OpenSearch Service.

To confirm the change knowledge seize (CDC), run the next command to replace the OnHand and MinOnHand fields for the prevailing doc merchandise Extremely GelPen within the product assortment:

db.product.updateOne(
   "Merchandise":"Extremely GelPen"
,

   "$set":
      "Stock":
         "OnHand":300,
         "MinOnHand":100
      
   
);

Confirm the CDC for the replace to the doc for the merchandise Extremely GelPen on the OpenSearch Service index.

Monitor the CDC pipeline

You possibly can monitor the state of the pipelines by checking the standing of the pipeline on the OpenSearch Service console. Moreover, you should use Amazon CloudWatch to supply real-time metrics and logs, which helps you to arrange alerts in case of a breach of user-defined thresholds.

Clear up

Be sure to clear up undesirable AWS sources created throughout this put up with a purpose to stop extra billing for these sources. Observe these steps to wash up your AWS account:

1. On the OpenSearch Service console, select Domains underneath Managed clusters within the navigation pane.
2. Choose the area you wish to delete and select Delete.
3. Select Pipelines underneath Ingestion within the navigation pane.
4. Choose the pipeline you wish to delete and on the Actions menu, select Delete.
5. On the Amazon S3 console, choose the S3 bucket and select Delete.

Conclusion

On this put up, you discovered allow zero-ETL integration between Amazon DocumentDB change knowledge streams and OpenSearch Service. To be taught extra about zero-ETL integrations obtainable with different knowledge sources, see Working with Amazon OpenSearch Ingestion pipeline integrations.

In regards to the Authors

Praveen Kadipikonda is a Senior Analytics Specialist Options Architect at AWS primarily based out of Dallas. He helps prospects construct environment friendly, performant, and scalable analytic options. He has labored with constructing databases and knowledge warehouse options for over 15 years.

Kaarthiik Thota is a Senior Amazon DocumentDB Specialist Options Architect at AWS primarily based out of London. He’s enthusiastic about database applied sciences and enjoys serving to prospects clear up issues and modernize functions utilizing NoSQL databases. Earlier than becoming a member of AWS, he labored extensively with relational databases, NoSQL databases, and enterprise intelligence applied sciences for over 15 years.

Muthu Pitchaimani is a Search Specialist with Amazon OpenSearch Service. He builds large-scale search functions and options. Muthu is within the subjects o f networking and safety, and relies out of Austin, Texas.