Import System

Data import mechanisms, formats, and validation rules

Overview

The Importer class processes JSON or stream data, handling entities, fields, relations, and various import options. Import data is structured as an object where each key is an entity name (e.g., products, users), and the value is an array of items to import for that entity.

Key Features

Entity-Based Structure Data organized by entity types with flexible operations and field mapping

Expression Support Dynamic calculations, variable interpolation, and complex transformations

Atomic Operations All-or-nothing imports with comprehensive error handling and rollback

Complete Import Example

{
  "@atomic": "batch",
  "@requireChangeSet": true,
  "products": [
    {
      "@operation": "upsert",
      "code": "WIDGET-001",
      "name": "Smart Widget",
      "basePrice": 100,
      "vatRate": 0.21,
      "price": { "e": "${basePrice} * (1 + ${vatRate})" },
      "category": { "e": "code:${categoryCode}" },
      "categoryCode": "ELECTRONICS", 
      "description": { "e": "${name} - Premium quality widget" },
      "validFrom": "today",
      "validTo": { "e": "${validFrom} + 1y" },
      "attributes": [
        {
          "key": "weight",
          "value": { "e": "${baseWeight} + 0.1" },
          "baseWeight": 2.5
        }
      ],
      "@action": "sync"
    }
  ],
  "categories": [
    {
      "@operation": "update",
      "@where": "status = 'active'",
      "status": "archived",
      "archivedAt": "now()"
    }
  ]
}

This example demonstrates:

‘Batch’ atomicity import with changeset tracking
Dynamic price calculation using expressions
Category lookup with variables
String interpolation for descriptions
Date arithmetic for validity periods
Nested expressions in multi-relations
Mass operations with @where conditions
Server action execution after import

Supported Field Types

String

Text values with expression support

Examples:

"name": "Widget"
"description": { "e": "Hello ${world}" }

Features:

Direct string values
Variable expressions
Templating

Number/Integer

Numeric values with calculator support

Examples:

"price": 123.45
"amount": "100+200"
"discount": "3.5%"
"total": { "e": "${price} * 1.21" }

Features:

Arithmetic operations
Percentages
Functions (abs, sqrt, etc.)
Variable interpolation

Boolean

True/false values with multiple formats

Examples:

"active": true
"enabled": "yes"
"visible": "ano"

Features:

Multiple language support
Various formats (1/0, yes/no)
Case insensitive

Date/Datetime

Date values with arithmetic support

Examples:

"date": "2024-06-26"
"date": "1.1.2025"
"dueDate": { "e": "${createdDate} + 30d" }
"validTo": "today + 1y"

Features:

Multiple formats
Relative expressions
Date arithmetic
Czech/European formats

Enum

Enumerated values with localization

Examples:

"status": "active"
"priority": "Vysoká"

Features:

Key or localized name
Case insensitive
Alias support

Translatable

Multi-language fields with three operation modes and translation control

Examples:

"name": "Value" (merge mode)
"name": {"en": "English", "cs": "Czech"} (replace mode)
"name": [{"language": "en", "value": "English"}] (array mode)
"name@removeAll": false (preserve other translations)

Features:

Language-specific update (merge)
Associative array (replace)
Array format (replace)
@removeAll control
@language directive support

Binary/Blob

Binary data with multiple encoding formats

Examples:

"content": { "base64": "SGVsbG8gV29ybGQ=" }
"content": { "hex": "48656c6c6f20576f726c64" }
"content": { "gzip-base64": "H4sIAAAAAAAA..." }
"content": "SGVsbG8=", "content@encoding": "base64"

Features:

Base64 encoding
Hexadecimal encoding
Gzip compression
Encoding suffix support
Object and string formats

Import Operations

@operation: create

Always create a new record

Example:

{ "@operation": "create", "name": "New Product" }

@operation: update

Update an existing record (requires primary key)

Example:

{ "@operation": "update", "id": 123, "name": "Updated Product" }

@operation: upsert

Create if not found, otherwise update (default)

Example:

{ "@operation": "upsert", "code": "ABC", "name": "Product ABC" }

@operation: delete

Delete the specified record(s)

Example:

{ "@operation": "delete", "id": 123 }

Upsert Behavior Control (@update and @create)

These directives provide fine-grained control over upsert behavior by specifying what should happen when a record exists (@update) or doesn’t exist (@create). They are typically used with @operation: “upsert”.

@update

Controls what happens when a record already exists during an upsert operation.

Values:

ok - Update the existing record (default)
ignore - Skip updating, leave unchanged
fail - Throw error if record exists

@create

Controls what happens when a record doesn’t exist during an upsert operation.

Values:

ok - Create a new record (default)
ignore - Skip creating, no new record
fail - Throw error if record doesn’t exist

Examples

Skip Updates for Existing Records

{
  "products": [
    {
      "@operation": "upsert",
      "@update": "ignore",
      "code": "EXISTING-PRODUCT",
      "name": "Updated Name"
    }
  ]
}

If a product with code “EXISTING-PRODUCT” already exists, it will be skipped (not updated).

Prevent Creation of New Records

{
  "users": [
    {
      "@operation": "upsert",
      "@create": "fail",
      "email": "user@example.com",
      "status": "active"
    }
  ]
}

If no user with the given email exists, the import will fail with an error instead of creating a new user.

Only Create New Records

{
  "categories": [
    {
      "@operation": "upsert",
      "@update": "fail",
      "@create": "ok",
      "code": "NEW-CATEGORY",
      "name": "New Category"
    }
  ]
}

This will create the category if it doesn’t exist, but fail if it already exists (ensuring only new categories are created).

Key Points

Default behavior is “ok” for both @update and @create
These directives only work with @operation: “upsert”
“ignore” skips the operation silently
“fail” throws an error and can break atomic imports
Useful for data synchronization and preventing overwrites

Transaction Modes (@atomic)

The @atomic directive controls transaction behavior and error handling strategy during import. Choose the appropriate mode based on your data size, error tolerance, and performance requirements.

@atomic: true (Default)

All imports are atomic - either the whole file commits or none. Most secure but can be slower for large imports.

Characteristics:

✓ All-or-nothing - Complete transaction integrity
✓ Rollback on error - Any error cancels all changes
⚠ Limited errors - Maximum 100 errors reported

@atomic: false

Each element is imported independently. Partial imports are possible - some records may succeed while others fail.

Characteristics:

✓ Independent records - Each record in its own transaction
✓ Partial success - Import as much data as possible
⚠ Error isolation - Errors don’t affect other records

@atomic: “batch” (Recommended)

The fastest and most effective mode. Imports are handled in batches for optimal performance while maintaining reasonable error isolation.

Characteristics:

🚀 High performance - Optimized for large datasets
⚖️ Balanced approach - Performance with error isolation
📦 Batch rollback - Failed batch doesn’t affect others

Examples

Strict All-or-Nothing Import

{
  "@atomic": true,
  "products": [
    { "code": "P001", "name": "Product 1" },
    { "code": "P002", "name": "Product 2" },
    { "code": "INVALID", "name": "" }
  ]
}

If any product fails validation, none of the products will be imported. All changes are rolled back.

Import Individual Records

{
  "@atomic": false,
  "products": [
    { "code": "P001", "name": "Product 1" },
    { "code": "P002", "name": "Product 2" },
    { "code": "INVALID", "name": "" }
  ]
}

Valid products (P001, P002) will be imported successfully, while the invalid one will be skipped with an error report.

Optimized Batch Processing

{
  "@atomic": "batch",
  "products": [
    { "code": "P001", "name": "Product 1" },
    { "code": "P002", "name": "Product 2" },
    { "code": "P003", "name": "Product 3" }
  ]
}

Products are processed in optimized batches. If one batch fails, other batches can still succeed, providing the best balance of performance and reliability.

Choosing the Right Mode

Use true for critical data where any error should stop the import
Use false for best-effort imports where partial success is acceptable
Use “batch” for large datasets requiring optimal performance
Default is true for maximum data integrity
Consider your error handling strategy and performance requirements

Batch Size (@batchSize)

The @batchSize directive overrides the number of items processed in a single database transaction. The default is 50; the maximum allowed value is 1000 (higher values are silently capped to prevent out-of-memory errors). Zero and negative values are rejected with an error.

This directive has no effect when @atomic: false is used, because each item is always wrapped in its own individual transaction.

Example

{
  "@atomic": "batch",
  "@batchSize": 200,
  "products": [
    { "code": "P001", "name": "Product 1" },
    { "code": "P002", "name": "Product 2" }
  ]
}

Products will be committed 200 at a time within a single transaction.

When to tune the default

Increase (e.g. 200–500) when importing simple, flat entities with no nested relations to improve throughput
Decrease (e.g. 10–20) when importing entities with many MANY relations or heavy automations to reduce memory pressure
The default of 50 is a good balance for most use cases

Automation Execution (@automation)

Execute predefined automations to apply complex transformations and business logic during import

The @automation directive allows you to execute predefined automations on imported records. This enables you to apply complex transformations, calculations, or business logic that is defined in automation entities.

How It Works

Record Selection - Records are selected using @where or identified by ID
Automation Lookup - The automation is retrieved by its ID or lookup key (code:)
Script Execution - The automation’s importScript is parsed and applied to each matching record
Field Updates - Changes defined in the automation are applied to the record
Save - The updated record is saved to the database

Examples

Basic Automation Execution

{
  "products": [
    {
      "@where": "price < 100",
      "@automation": "code:price-calculation-automation"
    }
  ]
}

Execute an automation on all products where price is less than 100.

Price Calculation Automation

Step 1: Create the automation

{
  "automations": [
    {
      "@operation": "create",
      "name": "Apply Standard Markup",
      "code": "price-calculation-automation",
      "type": "massUpdate",
      "importScript": "{\"products\":[{\"price\":{\"e\":\"${cost} * 1.3\"}}]}"
    }
  ]
}

Step 2: Apply the automation

{
  "products": [
    {
      "@where": "category = 'electronics'",
      "@automation": "Apply Standard Markup"
    }
  ]
}

Conditional Field Updates

{
  "automations": [
    {
      "name": "Low Stock Alert",
      "type": "massUpdate",
      "importScript": "{\"products\":[{\"@where\":\"quantity < 10\",\"status\":\"low-stock\",\"alertSent\":true}]}"
    }
  ]
}

// Apply to products
{
  "products": [
    {
      "@where": "quantity < 10",
      "@automation": "Low Stock Alert"
    }
  ]
}

The automation includes its own @where condition to double-check criteria.

Combining with Other Directives

{
  "products": [
    {
      "@where": "status = 'pending'",
      "@automation": "code:calculate-totals",
      "@action": "approve"
    }
  ]
}

Execution order: 1) Record located, 2) Automation executed, 3) Server action invoked.

Best Practices

Use Descriptive Names: Name automations clearly to indicate their purpose
Test Thoroughly: Test on sample data before production
Avoid Self-References: Never create automations that reference themselves
Use @where Conditions: Ensure automations only apply to appropriate records
Monitor Depth: Keep automation nesting shallow (max 5 levels)
Document Logic: Document complex automation logic

Troubleshooting

Automation not executing

Verify automation ID/lookup key is correct
Check automation type is massUpdate or aiMassUpdate
Ensure automation is not disabled
Verify importScript field is valid JSON

Automation depth exceeded error

Check for circular automation references
Reduce automation nesting level
Review automation chain for loops

Changes not applied

Verify @where condition in importScript matches record
Check field names in automation script are correct
Ensure importScript is properly formatted

Relations and Lookups

By Primary Key

Reference using the related entity’s primary key

"category": 123

By Lookup

Reference using lookup type and value

"category": "code:ELECTRONICS"

By External ID (ext:)

Reference using external identifiers stored in extIds (supports colons in value). External IDs are defined as key-value pairs in the extIds field of an entity, allowing flexible referencing without relying on internal IDs. We recommend using a prefix like ext:SYSTEM: to avoid conflicts with other integrations.

"category": "ext:SYSTEM:CATEGORY:123"

Using Expressions

Dynamic lookup using variables

"category": { "e": "code:${categoryCode}" }

Create with Additional Fields

Auto-create related entity with multiple fields using @create-as

"category_id": "code:ELEC",
"category_id@if-not-found": "create",
"category_id@create-as": {
  "name": "Electronics",
  "code": "ELEC"
}

Multiple Values

Try multiple resolution methods

"category": [123, "code:FALLBACK"]

Multi-Relations

One-to-many relationship arrays

"items": [
  { "name": "Item 1", "quantity": 10 },
  { "name": "Item 2", "quantity": 5 }
]

Special Field Suffixes

@atomic

Controls transaction behavior and error handling strategy

Values:

true (all-or-nothing)
false (independent records)
batch (batched processing)

@operation

Controls the action for the item

Values:

create
update
upsert
delete

@where

Defines conditions for mass operations

Values:

Object format
Expression format

@m

Specifies the author/source of change

Values:

api/a
human/user/u
default/d
import/i
ai

@if-not-found

Controls behavior if a lookup fails

Values:

fail
null
create
skip

@create-as

Specifies additional fields when creating a related entity with @if-not-found=create

Values:

Object with field names and values

@action

Invokes a server action on the imported record

Values:

merge
sync
custom actions

@automation

Executes a predefined automation on the imported record

Values:

automation ID
automation name/lookup

@batchSize

Sets the number of items processed in a single transaction (default: 50, maximum: 1000)

Values:

Positive integer (1–1000)

@requireChangeSet

Requires changes to be tracked within a changeset

Values:

true
false

@removeAll

Removes all existing related items not in import

Values:

true
false

@update

Controls behavior when record exists during upsert

Values:

ok
ignore
fail

@create

Controls behavior when record doesn’t exist during upsert

Values:

ok
ignore
fail

String Expressions

Powerful template language for dynamic string generation with variables, functions, and conditional logic

Basic Variable Interpolation

Use ${variableName} syntax to insert variable values. Variables are resolved from the current import context.

Examples:

"name": "Hello ${firstName}!" → "Hello John!"
"description": "${brand} - ${model}" → "Apple - iPhone"
"fullName": "${UPPER(firstName)} ${lastName}" → "JOHN Doe"

String Functions

Text Manipulation

UPPER(text) - Convert to uppercase
LOWER(text) - Convert to lowercase
TRIM(text) - Remove whitespace at both ends
LTRIM(text) - Remove whitespace from left end
RTRIM(text) - Remove whitespace from right end
UPPERFIRST(text) - Capitalize first letter
REPLACE(text, search, replacement) - Replace text
SUBSTR(text, start, length) - Extract substring

String Information & Utilities

LEN(text) - Get string length
STRIPHTML(html) - Remove HTML tags
ADDSLASHES(text) - Escape quotes
XMLESCAPE(text) - Escape XML chars
COALESCE(val1, val2, ...) - First non-null/non-empty value
ENCODE/DECODE(text, format) - Encoding operations (supported formats: base64, hex, base64url)

Regular Expression Functions

Use regular expressions for pattern matching and text extraction within string expressions.

Pattern Matching

REGEXPMATCH(pattern, text)

Test if text matches pattern. Returns true if the pattern matches the text, false otherwise.

Examples:

"${REGEXPMATCH('^[A-Z]', 'Hello')}" → true (starts with capital)
"${REGEXPMATCH('^[0-9]+$', 'Hello')}" → false (not all digits)

Text Extraction

REGEXP(pattern, text)

Extract text using regex. Returns first capturing group if present, otherwise full match. Returns null if no match.

Examples:

"${REGEXP('^([A-Z])[a-z]+', 'Hello')}" → "H" (first capturing group)
"${REGEXP('^[A-Z][a-z]+', 'Hello')}" → "Hello" (full match, no groups)

Common Use Cases

Email Validation

"${REGEXPMATCH('^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$', email)}"

Phone Number Extraction

"${REGEXP('\\(([0-9]{3})\\) [0-9]{3}-[0-9]{4}', '(555) 123-4567')}" → "555"

Data Validation

"${IF(REGEXPMATCH('^[A-Z]{2}[0-9]{4}$', code), 'Valid format', 'Invalid format')}"

Price Extraction

"${REGEXP('Price: \\$([0-9]+\\.[0-9]{2})', 'Price: $19.99')}" → "19.99"

Conditional Logic

IF Function

Use IF(condition, trueValue, falseValue) for conditional value selection.

Examples:

"status": "${IF(age >= 18, 'Adult', 'Minor')}" → "Adult"
"message": "${IF(isActive = true, 'Available', 'Unavailable')}" → "Available"

Ternary Operator

Alternative conditional syntax: condition ? trueValue : falseValue (same as IF function but more concise)

Examples:

"status": "${age >= 18 ? 'Adult' : 'Minor'}" → "Adult"
"message": "${isActive ? 'Available' : 'Unavailable'}" → "Available"

Nested Ternary Operators

Ternary operators can be nested for complex conditional logic.

Examples:

"category": "${age > 65 ? 'Senior' : (age > 18 ? 'Adult' : 'Minor')}" → "Adult"
"grade": "${score >= 90 ? 'A' : (score >= 80 ? 'B' : (score >= 70 ? 'C' : 'F'))}" → "A"
"access": "${userRole = 'admin' ? 'Full Access' : (userRole = 'user' ? 'Limited Access' : 'No Access')}" → "Full Access"

Ternary with Logical Operators

Combine ternary operators with logical operators for complex conditions.

Examples:

"eligibility": "${age >= 18 && isActive ? 'eligible' : 'not eligible'}" → "eligible"
"access": "${isAdmin || isManager ? 'access granted' : 'access denied'}" → "access granted"
"welcome": "${(age > 18 && status = 'verified') || isVIP ? 'Welcome' : 'Access denied'}" → "Welcome"

Operator Precedence

Precedence (highest to lowest)

Function calls
Parentheses grouping
Multiplication and Division (*, /)
Addition and Subtraction (+, -)
String concatenation (.)
Comparison operators
Logical AND (&&)
Logical OR (||)
Ternary conditional (? :)

Advanced Examples

Complex Conditional Logic

{
  "userStatus": {
    "e": "${(age >= 18 && isVerified = true) || isAdmin = true ? 'Authorized' : 'Unauthorized'}"
  },
  "priceCategory": {
    "e": "${price > 1000 ? 'Premium' : price > 500 ? 'Standard' : 'Budget'}"
  },
  "statusMessage": {
    "e": "${isActive ? (lastLogin ? 'Active User' : 'Inactive User') : 'Disabled Account'}"
  }
}

Multi-Level Function Nesting

{
  "processedName": {
    "e": "${UPPER(TRIM(REPLACE(rawName, '_', ' ')))}"
  }
}

Conditional String Building

{
  "fullAddress": {
    "e": "${street}${IF(LEN(apartment) > 0, ', Apt ' + apartment, '')}${IF(LEN(city) > 0, ', ' + city, '')}"
  }
}

Validation and Formatting

{
  "emailStatus": {
    "e": "${IF(LEN(email) > 0 && email != null, 'Email: ' + LOWER(TRIM(email)), 'No email provided')}"
  }
}

Complex Business Logic

{
  "accessLevel": {
    "e": "${IF((userRole = 'admin' || userRole = 'moderator') && isActive = true && LEN(lastLogin) > 0, 'Full Access', IF(userRole = 'user' && isActive = true, 'Limited Access', 'No Access'))}"
  }
}

Data Cleaning and Transformation

{
  "cleanDescription": {
    "e": "${TRIM(STRIPHTML(REPLACE(rawDescription, '\\n', ' ')))}"
  }
}

Quick Reference Examples

Basic Operations

"Hello ${world}!"
"${UPPER('hello')} ${LOWER('WORLD')}"
"${IF(isActive, 'Active', 'Inactive')}"
"${isActive ? 'Active' : 'Inactive'}"

Complex Conditions

"${IF(age >= 18 && status = 'verified', 'Welcome', 'Access denied')}"
"${price > 100 ? 'Premium' : price > 50 ? 'Standard' : 'Budget'}"
"${COALESCE(nickname, firstName, 'Anonymous')}"

Expression Format Examples

Any field can use an expression format by providing an object with an “e” property containing the expression string.

Number Calculator Examples

"amount": "100+200" → 300
"amount": "3.5%" → 0.035
"amount": "abs(-100)" → 100
"amount": "(100+50)*2" → 300
"amount": "sqrt(16)" → 4

Date Calculator Examples

"date": "1.1.2025" → 2025-01-01
"date": "15.6.25" → 2025-06-15
"date": "today + 1y" → one year from today
"date": "1.1.2025 + 5M" → 2025-06-01

Variable Interpolation

"price": { "e": "${basePrice} * 1.21" }
"fullName": { "e": "Hello ${firstName}!" }
"dueDate": { "e": "${createdDate} + 30d" }

Translatable Fields Update Modes

Comprehensive guide to handling multi-language fields with different update strategies

Translatable fields support multiple update modes with different behaviors for handling existing translations.

Three Update Modes

Mode 1: Language-specific update (merge mode)

"name": "Value"

Updates translation for the currently selected language only (@language or system default)

Uses: @language setting at top of import file (or system default language)
Default: "name@removeAll": false (preserves other language translations)

Example: If current language is “en” and existing translations are {"en": "Old", "cs": "Starý"}, then "name": "New" results in {"en": "New", "cs": "Starý"}

Mode 2: Direct associative array (replace mode)

"name": {"en": "English", "cs": "Czech"}

Replaces all translations with specified languages

Default: "name@removeAll": true (removes any other existing translations not specified)

Example: If existing translations are {"en": "Old", "cs": "Starý", "de": "Alt"}, then "name": {"en": "New", "fr": "Nouveau"} results in {"en": "New", "fr": "Nouveau"} (German translation is removed)

Mode 3: Direct array format (replace mode)

"name": [{"language": "en", "value": "English"}, {"language": "cs", "value": "Czech"}]
"name": [{"language": "en", "name": "English"}, {"language": "cs", "name": "Czech"}]

Alternative format supporting both “value” and “name” properties

Default: "name@removeAll": true (removes any other existing translations not specified)

Example: "name": [{"language": "en", "value": "New"}] removes all other language translations and keeps only English

Translation Removal Control

@removeAll: true

Remove all other translations not specified in current update

Warning: This will permanently delete unspecified translations

@removeAll: false

Preserve existing translations, only update/add specified ones

Safe: Existing translations are preserved and merged

Default Behavior Summary

Mode 1 (language-specific): @removeAll: false (merge)
Mode 2 & 3 (direct arrays): @removeAll: true (replace)

Practical Examples

Merge with existing (preserves other languages)

{
  "@language": "en",
  "products": [
    {
      "code": "WIDGET-001",
      "name": "New English Name",
      "name@removeAll": false
    }
  ]
}

This will update only the English translation, preserving existing Czech, German, etc.

Replace all translations

{
  "products": [
    {
      "code": "WIDGET-001",
      "name": {"en": "English", "cs": "Czech"}
    }
  ]
}

Automatically sets @removeAll: true, removing any other language translations.

Array format with selective removal control

{
  "products": [
    {
      "code": "WIDGET-001",
      "name": [{"language": "en", "value": "English"}],
      "name@removeAll": false
    }
  ]
}

Override default to preserve other languages while using array format.

Best Practices

Use mode 1 for incremental translation updates
Use modes 2/3 for bulk translation imports from external systems
Always specify @removeAll explicitly when behavior is critical
Consider using @language directive for consistent language handling across import
Test translation behavior with sample data before bulk imports

Common Pitfalls

**Accidental translation loss:** Mode 2/3 default to @removeAll: true
**Language directive confusion:** Ensure @language is set correctly for mode 1
**Mixed mode usage:** Be consistent within single import operation
**Case sensitivity:** Language codes are case-sensitive