path mgmt/v1alpha1/transformer.proto
package mgmt.v1alpha1
Messages
Name | Type | Description |
---|
account_id | string | The unique identifier of the account to create the user defined transformer for. |
name | string | The name of the user defined transformer. |
description | string | The description of the user defined transformer. |
source | TransformerSource | The source of the user defined transformer. |
transformer_config | TransformerConfig | The configuration of the user defined transformer. |
Name | Type | Description |
---|
transformer_id | string | The unique identifier of the user defined transformer to delete. |
GenerateBool
GenerateBusinessName
GenerateCardNumber
Name | Type | Description |
---|
valid_luhn | optional bool | Whether or not to validate the card number using the Luhn algorithm. |
GenerateCategorical
Name | Type | Description |
---|
categories | optional string | A comma separated list of categories that will be used to generate a random value from. |
GenerateCity
GenerateCountry
Name | Type | Description |
---|
generate_full_name | optional bool | An option to return the full country name of the randomly selected country or return the default of a 2-letter country code. |
GenerateDefault
GenerateE164PhoneNumber
Name | Type | Description |
---|
min | optional int64 | The minimum length of the phone number to generate. |
max | optional int64 | The maximum length of the phone number to generate. |
GenerateEmail
Name | Type | Description |
---|
email_type | optional GenerateEmailType | Optionally specify the type of email to generate. The types specified determine the contents on the left side of the @. |
GenerateFirstName
GenerateFloat64
Name | Type | Description |
---|
randomize_sign | optional bool | Whether or not to randomize the sign of the float64. |
min | optional double | The minimum value to randomize the float64 to. |
max | optional double | The maximum value to randomize the float64 to. |
precision | optional int64 | The precision of the float64 to generate. |
GenerateFullAddress
GenerateFullName
GenerateGender
Name | Type | Description |
---|
abbreviate | optional bool | Whether or not to abbreviate the gender. |
GenerateInt64
Name | Type | Description |
---|
randomize_sign | optional bool | Whether or not to randomize the sign of the int64. |
min | optional int64 | The minimum value to randomize the int64 to. |
max | optional int64 | The maximum value to randomize the int64 to. |
GenerateInt64PhoneNumber
GenerateIpAddress
GenerateJavascript
Name | Type | Description |
---|
code | string | The user provided javascript code that will be executed in the transformer |
GenerateLastName
GenerateSSN
GenerateSha256Hash
GenerateState
Name | Type | Description |
---|
generate_full_name | optional bool | An option to return the full state name of the randomly selected state or return the default of a 2-letter state code. |
GenerateStreetAddress
GenerateString
Name | Type | Description |
---|
min | optional int64 | The minimum length of the string to generate. |
max | optional int64 | The maximum length of the string to generate. |
GenerateStringPhoneNumber
Name | Type | Description |
---|
min | optional int64 | The minimum length of the phone number to generate. |
max | optional int64 | The maximum length of the phone number to generate. |
GenerateUnixTimestamp
GenerateUsername
GenerateUtcTimestamp
GenerateUuid
Name | Type | Description |
---|
include_hyphens | optional bool | Whether or not to include hyphens in the uuid. Defaults to true. If false, the uuid will be returned without hyphens. |
GenerateZipcode
Name | Type | Description |
---|
account_id | string | The unique identifier of the account to return entities for. |
Name | Type | Description |
---|
entities | repeated string | The list of built-in entities that are enabled for the provided account id. |
Name | Type | Description |
---|
transformer_id | string | The unique identifier of the user defined transformer to get. |
Name | Type | Description |
---|
account_id | string | The unique identifier of the account to get the user defined transformers for. |
Name | Type | Description |
---|
account_id | string | The unique identifier of the account to check for the availability of the transformer name. |
transformer_name | string | The name of the transformer to check for availability. |
Name | Type | Description |
---|
is_available | bool | Whether or not the transformer name is available. |
Null
Passthrough
PiiAnonymizer
PiiAnonymizer.Hash
PiiAnonymizer.Mask
Name | Type | Description |
---|
masking_char | optional string | Provide a character that will be used for the replacement. |
chars_to_mask | optional int32 | How many characters to mask. |
from_end | optional bool | Whether to mask the PII from start or end |
PiiAnonymizer.Redact
PiiAnonymizer.Replace
Name | Type | Description |
---|
value | optional string | The value to replace. If not provided, a template token of the anonymizer is used (e.g. A PERSON entity is replaced with: <PERSON>) |
Name | Type | Description |
---|
config | TransformerConfig | The transformer to use. If not provided, a transformer will automatically be selected (if supported), otherwise it falls back to generating a random hash. |
PiiDenyRecognizer
Name | Type | Description |
---|
name | string | Friendly name of this entity |
deny_words | repeated string | List of words that will be treated as PII. |
A system transformer is a transformer that is built into the Neosync platform.
It can be used directly in job mappings or used in user-defined transformers to snapshot a specific configuration.
Name | Type | Description |
---|
user_provided_regex | optional string | A user provided regular expression that will be used to scramble the characters in the input string. Must be a valid Golang regex. |
Name | Type | Description |
---|
preserve_length | optional bool | Whether or not to preserve the original length of the phone number. This causes the transformed phone number to retain the original length. |
Name | Type | Description |
---|
preserve_domain | optional bool | Whether or not to preserve the original domain, barring what has been specified in the excluded_domains property. |
preserve_length | optional bool | Whether or not to preserve the original length of the email. This causes the transformed email to retain the original length. |
excluded_domains | repeated string | A lsit of email domains that should be excluded. This changes based on the preserve_domain flag. See the docs for more details. |
email_type | optional GenerateEmailType | Optionally specify the type of email to generate. The types specified determine the contents on the left side of the @. |
invalid_email_action | optional InvalidEmailAction | Optionally specify an action in the result of an invalid email. An invalid email is one that does not conform to RFC 5322. |
Name | Type | Description |
---|
preserve_length | optional bool | Whether or not to preserve the original length of the first name. This causes the transformed first name to retain the original length. |
Name | Type | Description |
---|
randomization_range_min | optional double | The minimum value to randomize the float64 to. This will be relative to the original value. |
randomization_range_max | optional double | The maximum value to randomize the float64 to. This will be relative to the original value. |
Name | Type | Description |
---|
preserve_length | optional bool | Whether or not to preserve the original length of the full name. This causes the transformed full name to retain the original length. |
TransformHash is a transformer that hashes the input string using the specified hashing algorithm.
Name | Type | Description |
---|
randomization_range_min | optional int64 | The minimum value to randomize the int64 to. This will be relative to the original value. |
randomization_range_max | optional int64 | The maximum value to randomize the int64 to. This will be relative to the original value. |
Name | Type | Description |
---|
preserve_length | optional bool | Whether or not to preserve the original length of the phone number. This causes the transformed phone number to retain the original length. |
Name | Type | Description |
---|
code | string | The user provided javascript code that will be executed in the transformer |
Name | Type | Description |
---|
preserve_length | optional bool | Whether or not to preserve the original length of the last name. This causes the transformed last name to retain the original length. |
Name | Type | Description |
---|
preserve_length | optional bool | Whether or not to preserve the original length of the phone number. This causes the transformed phone number to retain the original length. |
TransformPiiText
NeosyncCloud/Enterprise only transformer for anonymizing PII Text
Name | Type | Description |
---|
score_threshold | float | Minimal detection score for determining PII. 0.0-1.0 |
default_anonymizer | PiiAnonymizer | The default anonmyization config used for all instances of detected PII. |
deny_recognizers | repeated PiiDenyRecognizer | Configure deny lists where each word is treated as PII. |
allowed_entities | repeated string | Configure a list of entities to be used for PII analysis. If not provided or empty, all entities are considiered
If this is specified, any ad-hoc, or deny_recognizers entity names must also be provided.
To see available builtin entities, call the GetPiiTextEntities() RPC method to see what is available for your account. |
allowed_phrases | repeated string | Exact match of PII phrases identified by the analysis engine. |
language | optional string | Two characters for the desired language in ISO_639-1 format.
If not provided, will use a default language if specified on the server.
Depending on the server configuration, may attempt to autodetect from input. |
entity_anonymizers | repeated TransformPiiText.EntityAnonymizersEntry | A map of entity names to anonymizer configurations.
The key in this map corresponds to a recognized entity (e.g. PERSON, PHONE_NUMBER) which can be found via the GetPiiTextEntities() RPC method.
The value in this map corresponds to the anonymizer configuration to use for that entity.
If not provided, the default_anonymizer will be used. |
TransformPiiText.EntityAnonymizersEntry
Takes an input integer and scrambles it, effectively outputing another integer that is not the same as the input.
This is effectively a random integer generator that has unique properties.
Name | Type | Description |
---|
preserve_length | optional bool | Whether or not to preserve the original length of the string. This causes the transformed string to retain the original length. |
The configuration for a transformer. This determines what type of transformation or generation is performed.
Name | Type | Description |
---|
transformer_id | string | The unique identifier of the user defined transformer to update. |
name | string | The name of the user defined transformer. |
description | string | The description of the user defined transformer. |
transformer_config | TransformerConfig | The configuration of the user defined transformer. |
Name | Type | Description |
---|
id | string | The unique identifier of the user defined transformer. |
name | string | The name of the user defined transformer. |
description | string | The description of the user defined transformer. |
data_type | TransformerDataType | @deprecated: data_type is deprecated and will be removed in a future release. Use data_types instead. |
source | TransformerSource | The source of the user defined transformer. |
config | TransformerConfig | The configuration of the user defined transformer. |
created_at | google.protobuf.Timestamp | The date and time the user defined transformer was created. |
updated_at | google.protobuf.Timestamp | The date and time the user defined transformer was last updated. |
account_id | string | The unique identifier of the account that owns the user defined transformer. |
data_types | repeated TransformerDataType | The data types supported by the user defined transformer. This is currently calculated based off the origin system transformer. |
Name | Type | Description |
---|
id | string | The id of the user defined transformer. When used in a job mapping, this will be used to reference the underlying system transformer along with it's configuration that was chosen. |
ValidateUserJavascriptCodeRequest
Name | Type | Description |
---|
code | string | The user provided javascript code that will be validated. |
ValidateUserJavascriptCodeResponse
Name | Type | Description |
---|
valid | bool | |
ValidateUserRegexCodeRequest
Name | Type | Description |
---|
user_provided_regex | string | The user provided regex to validate |
ValidateUserRegexCodeResponse
Name | Type | Description |
---|
valid | bool | Whether the regex is valid |
Enums
GenerateEmailType
Name | Number | Description |
---|
GENERATE_EMAIL_TYPE_UNSPECIFIED | 0 | Unspecified defaults to uuid v4 |
GENERATE_EMAIL_TYPE_UUID_V4 | 1 | Generates a random uuid v4 (without hyphens) on the left side of the @. This has all the uniqueness guarantees of the standard uuid v4, along with randomized domains. |
GENERATE_EMAIL_TYPE_FULLNAME | 2 | Generates a full name (firstnamelastname) on the left side of the @. This has high cardinality but is not guaranteed to be unique. |
GenerateIpAddressType
Name | Number | Description |
---|
GENERATE_IP_ADDRESS_TYPE_UNSPECIFIED | 0 | Unspecified defaults to ipv4 |
GENERATE_IP_ADDRESS_TYPE_V4_PUBLIC | 1 | Generates a random ipv4_public |
GENERATE_IP_ADDRESS_TYPE_V4_PRIVATE_A | 2 | Generates a random ipv4_private_a |
GENERATE_IP_ADDRESS_TYPE_V4_PRIVATE_B | 3 | Generates a random ipv4_private_b |
GENERATE_IP_ADDRESS_TYPE_V4_PRIVATE_C | 4 | Generates a random ipv4_private_c |
GENERATE_IP_ADDRESS_TYPE_V4_LINK_LOCAL | 5 | Generates a random ipv4_linklocal |
GENERATE_IP_ADDRESS_TYPE_V4_MULTICAST | 6 | Generates a random ipv4_multicast |
GENERATE_IP_ADDRESS_TYPE_V4_LOOPBACK | 7 | Generates a random ipv4_loopback |
GENERATE_IP_ADDRESS_TYPE_V6 | 8 | Generates a random ipv6 |
InvalidEmailAction
Action enumeration that dictates what the transform_email transformer should do in the case that it encounters an email that does not conform to RFC 5322
Name | Number | Description |
---|
INVALID_EMAIL_ACTION_UNSPECIFIED | 0 | Unspecified defaults to reject. |
INVALID_EMAIL_ACTION_REJECT | 1 | If an invalid email is encountered, the transformer will reject the transformation and exit. |
INVALID_EMAIL_ACTION_NULL | 2 | The result of the email will be null. This should only be specified if the destination column is also nullable. |
INVALID_EMAIL_ACTION_PASSTHROUGH | 3 | The result of the email will be identical to the original value, unmodified. |
INVALID_EMAIL_ACTION_GENERATE | 4 | The result of the email will be a brand new generated email. This is the same result as the generate_email transformer. |
PiiAnonymizer.Hash.HashType
Name | Number | Description |
---|
HASH_TYPE_UNSPECIFIED | 0 | If unspecified, defaults to md5 |
HASH_TYPE_MD5 | 1 | The MD5 Hash Algorithm |
HASH_TYPE_SHA256 | 2 | The SHA256 Hash Algorithm |
HASH_TYPE_SHA512 | 3 | The SHA512 Hash Algorithm |
SupportedJobType
Name | Number | Description |
---|
SUPPORTED_JOB_TYPE_UNSPECIFIED | 0 | |
SUPPORTED_JOB_TYPE_SYNC | 1 | Determines if a transformer is supported for a sync job. |
SUPPORTED_JOB_TYPE_GENERATE | 2 | Determines if a transformer is supported for a generate job. |
Name | Number | Description |
---|
HASH_TYPE_UNSPECIFIED | 0 | If unspecified, defaults to md5 |
HASH_TYPE_MD5 | 1 | The MD5 Hash Algorithm |
HASH_TYPE_SHA1 | 2 | The SHA1 Hash Algorithm |
HASH_TYPE_SHA256 | 3 | The SHA256 Hash Algorithm |
HASH_TYPE_SHA512 | 4 | The SHA512 Hash Algorithm |
Name | Number | Description |
---|
TRANSFORMER_DATA_TYPE_UNSPECIFIED | 0 | |
TRANSFORMER_DATA_TYPE_STRING | 1 | |
TRANSFORMER_DATA_TYPE_INT64 | 2 | |
TRANSFORMER_DATA_TYPE_BOOLEAN | 3 | |
TRANSFORMER_DATA_TYPE_FLOAT64 | 4 | |
TRANSFORMER_DATA_TYPE_NULL | 5 | |
TRANSFORMER_DATA_TYPE_ANY | 6 | |
TRANSFORMER_DATA_TYPE_TIME | 7 | |
TRANSFORMER_DATA_TYPE_UUID | 8 | |