SageMaker

Module sagemaker

Important Capabilities

Capability	Status	Notes
Table-Level Lineage	✅	Enabled by default

This plugin extracts the following:

• Feature groups
• Models, jobs, and lineage between the two (e.g. when jobs output a model or a model is used by a job)

CLI based Ingestion

Install the Plugin

pip install 'acryl-datahub[sagemaker]'

Starter Recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: sagemaker
  config:
    # Coordinates
    aws_region: "my-aws-region"

sink:
  # sink configs

Config Details

Options

Note that a . is used to denote nested fields in the YAML recipe.

View All Configuration Options

Field [Required]	Type	Description	Default	Notes
aws_access_key_id [✅]	string	AWS access key ID. Can be auto-detected, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html for details.	None
aws_endpoint_url [✅]	string	Autodetected. See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html	None
aws_profile [✅]	string	Named AWS profile to use. Only used if access key / secret are unset. If not set the default will be used	None
aws_proxy [✅]	map(str,string)		None
aws_region [✅]	string	AWS region code.	None
aws_secret_access_key [✅]	string	AWS secret access key. Can be auto-detected, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html for details.	None
aws_session_token [✅]	string	AWS session token. Can be auto-detected, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html for details.	None
extract_feature_groups [✅]	boolean	Whether to extract feature groups.	True
extract_jobs [✅]	UnionType (See notes for variants)	Whether to extract AutoML jobs.	True	One of string,boolean
extract_models [✅]	boolean	Whether to extract models.	True
env [✅]	string	The environment that all assets produced by this connector belong to	PROD
aws_role [✅]	UnionType (See notes for variants)	AWS roles to assume. If using the string format, the role ARN can be specified directly. If using the object format, the role can be specified in the RoleArn field and additional available arguments are documented at https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html?highlight=assume_role#STS.Client.assume_role	None	One of string,union(anyOf),string,AwsAssumeRoleConfig
aws_role.ExternalId [❓ (required if aws_role is set)]	string	External ID to use when assuming the role.	None
aws_role.RoleArn [❓ (required if aws_role is set)]	string	ARN of the role to assume.	None
database_pattern [✅]	AllowDenyPattern	regex patterns for databases to filter in ingestion.	{'allow': ['.*'], 'deny': [], 'ignoreCase': True}
database_pattern.allow [❓ (required if database_pattern is set)]	array(string)		None
database_pattern.deny [❓ (required if database_pattern is set)]	array(string)		None
database_pattern.ignoreCase [❓ (required if database_pattern is set)]	boolean	Whether to ignore case sensitivity during pattern matching.	True
table_pattern [✅]	AllowDenyPattern	regex patterns for tables to filter in ingestion.	{'allow': ['.*'], 'deny': [], 'ignoreCase': True}
table_pattern.allow [❓ (required if table_pattern is set)]	array(string)		None
table_pattern.deny [❓ (required if table_pattern is set)]	array(string)		None
table_pattern.ignoreCase [❓ (required if table_pattern is set)]	boolean	Whether to ignore case sensitivity during pattern matching.	True

Schema

The JSONSchema for this configuration is inlined below.

{
  "title": "SagemakerSourceConfig",
  "description": "Common AWS credentials config.\n\nCurrently used by:\n    - Glue source\n    - SageMaker source",
  "type": "object",
  "properties": {
    "aws_access_key_id": {
      "title": "Aws Access Key Id",
      "description": "AWS access key ID. Can be auto-detected, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html for details.",
      "type": "string"
    },
    "aws_secret_access_key": {
      "title": "Aws Secret Access Key",
      "description": "AWS secret access key. Can be auto-detected, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html for details.",
      "type": "string"
    },
    "aws_session_token": {
      "title": "Aws Session Token",
      "description": "AWS session token. Can be auto-detected, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html for details.",
      "type": "string"
    },
    "aws_role": {
      "title": "Aws Role",
      "description": "AWS roles to assume. If using the string format, the role ARN can be specified directly. If using the object format, the role can be specified in the RoleArn field and additional available arguments are documented at https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html?highlight=assume_role#STS.Client.assume_role",
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "array",
          "items": {
            "anyOf": [
              {
                "type": "string"
              },
              {
                "$ref": "#/definitions/AwsAssumeRoleConfig"
              }
            ]
          }
        }
      ]
    },
    "aws_profile": {
      "title": "Aws Profile",
      "description": "Named AWS profile to use. Only used if access key / secret are unset. If not set the default will be used",
      "type": "string"
    },
    "aws_region": {
      "title": "Aws Region",
      "description": "AWS region code.",
      "type": "string"
    },
    "aws_endpoint_url": {
      "title": "Aws Endpoint Url",
      "description": "Autodetected. See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html",
      "type": "string"
    },
    "aws_proxy": {
      "title": "Aws Proxy",
      "description": "Autodetected. See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html",
      "type": "object",
      "additionalProperties": {
        "type": "string"
      }
    },
    "env": {
      "title": "Env",
      "description": "The environment that all assets produced by this connector belong to",
      "default": "PROD",
      "type": "string"
    },
    "database_pattern": {
      "title": "Database Pattern",
      "description": "regex patterns for databases to filter in ingestion.",
      "default": {
        "allow": [
          ".*"
        ],
        "deny": [],
        "ignoreCase": true
      },
      "allOf": [
        {
          "$ref": "#/definitions/AllowDenyPattern"
        }
      ]
    },
    "table_pattern": {
      "title": "Table Pattern",
      "description": "regex patterns for tables to filter in ingestion.",
      "default": {
        "allow": [
          ".*"
        ],
        "deny": [],
        "ignoreCase": true
      },
      "allOf": [
        {
          "$ref": "#/definitions/AllowDenyPattern"
        }
      ]
    },
    "extract_feature_groups": {
      "title": "Extract Feature Groups",
      "description": "Whether to extract feature groups.",
      "default": true,
      "type": "boolean"
    },
    "extract_models": {
      "title": "Extract Models",
      "description": "Whether to extract models.",
      "default": true,
      "type": "boolean"
    },
    "extract_jobs": {
      "title": "Extract Jobs",
      "description": "Whether to extract AutoML jobs.",
      "default": true,
      "anyOf": [
        {
          "type": "object",
          "additionalProperties": {
            "type": "string"
          }
        },
        {
          "type": "boolean"
        }
      ]
    }
  },
  "required": [
    "aws_region"
  ],
  "additionalProperties": false,
  "definitions": {
    "AwsAssumeRoleConfig": {
      "title": "AwsAssumeRoleConfig",
      "type": "object",
      "properties": {
        "RoleArn": {
          "title": "Rolearn",
          "description": "ARN of the role to assume.",
          "type": "string"
        },
        "ExternalId": {
          "title": "Externalid",
          "description": "External ID to use when assuming the role.",
          "type": "string"
        }
      },
      "required": [
        "RoleArn"
      ]
    },
    "AllowDenyPattern": {
      "title": "AllowDenyPattern",
      "description": "A class to store allow deny regexes",
      "type": "object",
      "properties": {
        "allow": {
          "title": "Allow",
          "description": "List of regex patterns to include in ingestion",
          "default": [
            ".*"
          ],
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "deny": {
          "title": "Deny",
          "description": "List of regex patterns to exclude from ingestion.",
          "default": [],
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "ignoreCase": {
          "title": "Ignorecase",
          "description": "Whether to ignore case sensitivity during pattern matching.",
          "default": true,
          "type": "boolean"
        }
      },
      "additionalProperties": false
    }
  }
}

Code Coordinates

• Class Name: datahub.ingestion.source.aws.sagemaker.SagemakerSource
• Browse on GitHub

Questions

If you've got any questions on configuring ingestion for SageMaker, feel free to ping us on our Slack