This repository offers a comprehensive collection of code samples and tutorials for AWS Glue, a serverless data integration service. It empowers users to discover, prepare, move, and integrate data for analytics and machine learning workloads. The samples cater to engineers and data professionals seeking practical examples to implement various AWS Glue features, from basic ETL to advanced streaming and data quality management.

How It Works

The samples demonstrate core AWS Glue functionalities, including ETL scripting, data catalog management, crawlers, data quality checks, and streaming jobs. They showcase integration with modern data lake formats like Apache Hudi, Delta Lake, and Apache Iceberg, and leverage AWS Glue for Ray to scale Python and pandas workloads. The examples are designed to be executable within AWS Glue ETL jobs, containerized environments, or local development setups.

Quick Start & Requirements

• Samples can be run on AWS Glue ETL jobs, containers, or local environments.
• Local development is supported via Docker, with guides for Spark-submit, REPL shells, unit testing (pytest), JupyterLab, and VS Code.
• Core AWS Glue Python libraries are available in a separate repository: awslabs/aws-glue-libs.
• No specific hardware or OS prerequisites are detailed beyond standard AWS Glue/Spark environment requirements.

Highlighted Details

• Native support for Apache Hudi, Delta Lake, and Apache Iceberg on AWS Glue for Apache Spark.
• AWS Glue for Ray integration for scaling Python workloads and AWS SDK for pandas.
• Extensive AWS Glue Data Quality examples, covering setup, advanced rules, alerts, and visualization.
• CI/CD pipeline examples using AWS CDK for ETL solutions.
• Utilities for Hive metastore migration, crawler management, and Glue resource synchronization.
• Demonstrates Glue Custom Connectors for integrating with third-party data stores.

Maintenance & Community

No specific details regarding contributors, community channels, or roadmap were found in the provided README content.

Licensing & Compatibility

The sample code is provided under the MIT-0 license. This permissive license allows for broad usage, including commercial applications, with minimal restrictions.

Limitations & Caveats

No explicit limitations, known bugs, or alpha status were mentioned in the provided README content.