Streamline your information governance by deploying Amazon DataZone with the AWS CDK

Managing information throughout various environments is usually a advanced and daunting process. Amazon DataZone simplifies this so you possibly can catalog, uncover, share, and govern information saved throughout AWS, on premises, and third-party sources.

Many organizations handle huge quantities of knowledge property owned by numerous groups, creating a fancy panorama that poses challenges for scalable information administration. These organizations require a strong infrastructure as code (IaC) strategy to deploy and handle their information governance options. On this put up, we discover deploy Amazon DataZone utilizing the AWS Cloud Improvement Equipment (AWS CDK) to realize seamless, scalable, and safe information governance.

Overview of resolution

By utilizing IaC with the AWS CDK, organizations can effectively deploy and handle their information governance options. This strategy offers scalability, safety, and seamless integration throughout all groups, permitting for constant and automatic deployments.

The AWS CDK is a framework for outlining cloud IaC and provisioning it by AWS CloudFormation. Builders can use any of the supported programming languages to outline reusable cloud parts referred to as constructs. A assemble is a reusable and programmable element that represents AWS sources. The AWS CDK interprets the high-level constructs outlined by you into equal CloudFormation templates. AWS CloudFormation provisions the sources specified within the template, streamlining the utilization of IaC on AWS.

Amazon DataZone core parts are the constructing blocks to create a complete end-to-end resolution for information administration and information governance. The next are the Amazon DataZone core parts. For extra particulars, see Amazon DataZone terminology and ideas.

• Amazon DataZone area – You should use an Amazon DataZone area to arrange your property, customers, and their tasks. By associating further AWS accounts together with your Amazon DataZone domains, you possibly can convey collectively your information sources.
• Knowledge portal – The information portal is outdoors the AWS Administration Console. It is a browser-based net software the place completely different customers can catalog, uncover, govern, share, and analyze information in a self-service style.
• Enterprise information catalog – You should use this element to catalog information throughout your group with enterprise context and allow everybody in your group to find and perceive information shortly.
• Tasks – In Amazon DataZone, tasks are enterprise use case-based groupings of individuals, property (information), and instruments used to simplify entry to AWS analytics.
• Environments – Inside Amazon DataZone tasks, environments are collections of zero or extra configured sources on which a given set of AWS Identification and Entry Administration (IAM) principals (for instance, customers with a contributor permissions) can function.
• Amazon DataZone information supply – In Amazon DataZone, you possibly can publish an AWS Glue Knowledge Catalog information supply or Amazon Redshift information supply.
• Publish and subscribe workflows – You should use these automated workflows to safe information between producers and customers in a self-service method and be sure that everybody in your group has entry to the appropriate information for the appropriate objective.

We use an AWS CDK app to display create and deploy core parts of Amazon DataZone in an AWS account. The next diagram illustrates the first core parts that we create.

Along with the core parts deployed with the AWS CDK, we offer a customized useful resource module to create Amazon DataZone parts equivalent to glossaries, glossary phrases, and metadata kinds, which aren’t supported by AWS CDK constructs (on the time of writing).

Stipulations

The next native machine stipulations are required earlier than beginning:

• An AWS account (with AWS IAM Identification Heart enabled).
• Both Bash or ZSH terminal.
• The AWS Command Line Interface (AWS CLI) v2 put in.
• Python model 3.10 or increased.
• The AWS SDK for Python model 1.34.87 or increased.
• Node model v18.17.* or increased.
• NPM model v10.2.* or increased.
• An AWS Glue desk to be registered as a pattern information supply in an Amazon DataZone mission.
• As a part of this put up, we wish to publish AWS Glue tables from an AWS Glue database that already exists. For this, you need to explicitly present Amazon DataZone with the permissions to entry tables on this present AWS Glue database. For extra info, confer with Configure Lake Formation permissions for Amazon DataZone.

  

Deploy the answer

Full the next steps to deploy the answer:

1. Clone the GitHub repository and go to the basis of your downloaded repository folder:

   git clone https://github.com/aws-samples/amazon-datazone-cdk-example.git
   cd amazon-datazone-cdk-example

2. Set up native dependencies:

   $ npm ci ### this may set up the packages configured in package-lock.json

3. Sign up to your AWS account utilizing the AWS CLI by configuring your credential file (exchange <PROFILE_NAME> with the profile identify of your deployment AWS account):

   $ export AWS_PROFILE=<PROFILE_NAME>

4. Bootstrap the AWS CDK setting (this can be a one-time exercise and never wanted in case your AWS account is already bootstrapped):
5. Run the script to switch the placeholders in your AWS account and AWS Area within the config recordsdata:

   $ ./scripts/put together.sh <<YOUR_AWS_ACCOUNT_ID>> <<YOUR_AWS_REGION>>

The previous command will exchange the AWS_ACCOUNT_ID_PLACEHOLDER and AWS_REGION_PLACEHOLDER values within the following config recordsdata:

• lib/config/project_config.json
• lib/config/project_environment_config.json
• lib/constants.ts

Subsequent, you configure your Amazon DataZone area, mission, enterprise glossary, metadata kinds, and environments together with your information supply.

6. Go to the file lib/constants.ts. You possibly can preserve the DOMAIN_NAME supplied or replace it as wanted.
7. Go to the file lib/config/project_config.json. You possibly can preserve the instance values for projectName and projectDescription or replace them. An instance worth for projectMembers has additionally been supplied (as proven within the following code snippet). Replace the worth of the memberIdentifier parameter with an IAM function ARN of your selection that you just wish to be the proprietor of this mission.

   "projectMembers": [
               
                   "memberIdentifier": "arn:aws:iam::AWS_ACCOUNT_ID_PLACEHOLDER:role/Admin",
                   "memberIdentifierType": "UserIdentifier"
               
           ]

8. Go to the file lib/config/project_glossary_config.json. An instance enterprise glossary and glossary phrases are supplied for the tasks; you possibly can preserve them as is or replace them together with your mission identify, enterprise glossary, and glossary phrases.
9. Go to the lib/config/project_form_config.json file. You possibly can preserve the instance metadata kinds supplied for the tasks or replace your mission identify and metadata kinds.
10. Go to the lib/config/project_enviornment_config.json file. Replace EXISTING_GLUE_DB_NAME_PLACEHOLDER with the prevailing AWS Glue database identify in the identical AWS account the place you’re deploying the Amazon DataZone core parts with the AWS CDK. Be sure to have no less than one present AWS Glue desk on this AWS Glue database to publish as an information supply inside Amazon DataZone. Exchange DATA_SOURCE_NAME_PLACEHOLDER and DATA_SOURCE_DESCRIPTION_PLACEHOLDER together with your selection of Amazon DataZone information supply identify and outline. An instance of a cron schedule has been supplied (see the next code snippet). That is the schedule in your information supply run; you possibly can preserve the identical or replace it.

    "Schedule":
       "schedule":"cron(0 7 * * ? *)"
    

Subsequent, you replace the belief coverage of the AWS CDK deployment IAM function to deploy a customized useful resource module.

11. On the IAM console, replace the belief coverage of the IAM function in your AWS CDK deployment that begins with cdk-hnb659fds-cfn-exec-role- by including the next permissions. Exchange $ACCOUNT_ID and $REGION together with your particular AWS account and Area.

         
             "Impact": "Permit",
             "Principal": 
                 "Service": "lambda.amazonaws.com"
             ,
             "Motion": "sts:AssumeRole",
             "Situation": 
                 "ArnLike": 
                     "aws:SourceArn": [
                         
                         "arn:aws:lambda:$REGION:ACCOUNT_ID:function:DataZonePreqStack-GlossaryLambda*",
                         "arn:aws:lambda:$REGION:ACCOUNT_ID:function:DataZonePreqStack-GlossaryTermLambda*",
                         "arn:aws:lambda:$REGION:ACCOUNT_ID:function:DataZonePreqStack-FormLambda*"
                     ]
                 
             
         

Now you possibly can configure information lake directors in Lake Formation.

12. On the Lake Formation console, select Administrative roles and duties within the navigation pane.
13. Underneath Knowledge lake directors, select Add and add the IAM function for AWS CDK deployment that begins with cdk-hnb659fds-cfn-exec-role- as an administrator.

This IAM function wants permissions in Lake Formation to create sources, equivalent to an AWS Glue database. With out these permissions, the AWS CDK stack deployment will fail.

14. Deploy the answer:

    $ npm run cdk deploy --all

15. Throughout deployment, enter y if you wish to deploy the modifications for some stacks if you see the immediate Do you want to deploy these modifications (y/n)?.
16. After the deployment is full, register to your AWS account and navigate to the AWS CloudFormation console to confirm that the infrastructure deployed.

It’s best to see an inventory of the deployed CloudFormation stacks, as proven within the following screenshot.

17. Open the Amazon DataZone console in your AWS account and open your area.
18. Open the info portal URL obtainable within the Abstract part.
    
19. Discover your mission within the information portal and run the info supply job.

It is a one-time exercise if you wish to publish and search the info supply instantly inside Amazon DataZone. In any other case, anticipate the info supply runs based on the cron schedule talked about within the previous steps.

Troubleshooting

In case you get the message "Area identify already exists underneath this account, please use one other one (Service: DataZone, Standing Code: 409, Request ID: 2d054cb0-0 fb7-466f-ae04-c53ff3c57c9a)" (RequestToken: 85ab4aa7-9e22-c7e6-8f00-80b5871e4bf7, HandlerErrorCode: AlreadyExists), change the area identify underneath lib/constants.ts and attempt to deploy once more.

In case you get the message "Useful resource of kind 'AWS::IAM::Function' with identifier 'CustomResourceProviderRole1' already exists." (RequestToken: 17a6384e-7b0f-03b3 -1161-198fb044464d, HandlerErrorCode: AlreadyExists), this implies you’re by chance attempting to deploy every little thing in the identical account however a unique Area. Be certain that to make use of the Area you configured in your preliminary deployment. For the sake of simplicity, the DataZonePreReqStack is in a single Area in the identical account.

In case you get the message “Unmanaged asset” Warning within the information asset in your datazone mission, you need to explicitly present Amazon DataZone with Lake Formation permissions to entry tables on this exterior AWS Glue database. For directions, confer with Configure Lake Formation permissions for Amazon DataZone.

Clear up

To keep away from incurring future prices, delete the sources. If in case you have already shared the info supply utilizing Amazon DataZone, then you must take away these manually first within the Amazon DataZone information portal as a result of the AWS CDK isn’t capable of routinely do this.

1. Unpublish the info inside the Amazon DataZone information portal.
2. Delete the info asset from the Amazon DataZone information portal.
3. From the basis of your repository folder, run the next command:

   $ npm run cdk destroy --all

4. Delete the Amazon DataZone created databases in AWS Glue. Confer with the tricks to troubleshoot Lake Formation permission errors in AWS Glue if wanted.
   
5. Take away the created IAM roles from Lake Formation administrative roles and duties.
   

Conclusion

Amazon DataZone gives a complete resolution for implementing an information mesh structure, enabling organizations to deal with superior information governance challenges successfully. Utilizing the AWS CDK for IaC streamlines the deployment and administration of Amazon DataZone sources, selling consistency, reproducibility, and automation. This strategy enhances information group and sharing throughout your group.

Able to streamline your information governance? Dive deeper into Amazon DataZone by visiting the Amazon DataZone Consumer Information. To be taught extra concerning the AWS CDK, discover the AWS CDK Developer Information.

In regards to the Authors

Bandana Das is a Senior Knowledge Architect at Amazon Net Companies and focuses on information and analytics. She builds event-driven information architectures to assist clients in information administration and data-driven decision-making. She can be keen about enabling clients on their information administration journey to the cloud.

Gezim Musliaj is a Senior DevOps Marketing consultant with AWS Skilled Companies. He’s desirous about numerous issues CI/CD, information, and their software within the discipline of IoT, huge information ingestion, and lately MLOps and GenAI.

Sameer Ranjha is a Software program Improvement Engineer on the Amazon DataZone workforce. He works within the area of contemporary information architectures and software program engineering, creating scalable and environment friendly options.

Sindi Cali is an Affiliate Marketing consultant with AWS Skilled Companies. She helps clients in constructing data-driven functions in AWS.

Bhaskar Singh is a Software program Improvement Engineer on the Amazon DataZone workforce. He has contributed to implementing AWS CloudFormation assist for Amazon DataZone. He’s keen about distributed methods and devoted to fixing clients’ issues.