DynamoDB
- Availability
- Airbyte Cloud Airbyte OSS
- Support Level
- Community
- Latest Version
- 0.1.7
- Definition Id
- 8ccd8909-4e99-4141-b48d-4984b70b2d89
This destination writes data to AWS DynamoDB.
The Airbyte DynamoDB destination allows you to sync data to AWS DynamoDB. Each stream is written to its own table under the DynamoDB.
Prerequisites
- For Airbyte Open Source users using the Postgres source connector, upgrade your Airbyte platform to version
v0.40.0-alphaor newer and upgrade your DynamoDB connector to version0.1.5or newer
Sync overview
Output schema
Each stream will be output into its own DynamoDB table. Each table will a collections of json objects containing 4 fields:
_airbyte_ab_id: a uuid assigned by Airbyte to each event that is processed._airbyte_emitted_at: a timestamp representing when the event was pulled from the data source._airbyte_data: a json blob representing with the extracted data.sync_time: a timestamp representing when the sync up task be triggered.
Features
| Feature | Support | Notes |
|---|---|---|
| Full Refresh Sync | ✅ | Warning: this mode deletes all previously synced data in the configured DynamoDB table. |
| Incremental - Append Sync | ✅ | |
| Incremental - Append + Deduped | ❌ | |
| Namespaces | ✅ | Namespace will be used as part of the table name. |
Performance considerations
This connector by default uses 10 capacity units for both Read and Write in DynamoDB tables. Please provision more capacity units in the DynamoDB console when there are performance constraints.
Getting started
Requirements
- Allow connections from Airbyte server to your AWS DynamoDB tables (if they exist in separate VPCs).
- The credentials for AWS DynamoDB (for the COPY strategy).
Setup guide
- Fill up DynamoDB info
- DynamoDB Endpoint
- Leave empty if using AWS DynamoDB, fill in endpoint URL if using customized endpoint.
- DynamoDB Table Name
- The name prefix of the DynamoDB table to store the extracted data. The table name is \\.
- DynamoDB Region
- The region of the DynamoDB.
- Access Key Id
- See this on how to generate an access key.
- We recommend creating an Airbyte-specific user. This user will require read and write permissions to the DynamoDB table.
- Secret Access Key
- Corresponding key to the above key id.
- DynamoDB Endpoint
- Make sure your DynamoDB tables are accessible from the machine running Airbyte.
- This depends on your networking setup.
- You can check AWS DynamoDB documentation with a tutorial on how to properly configure your DynamoDB's access here.
- The easiest way to verify if Airbyte is able to connect to your DynamoDB tables is via the check connection tool in the UI.
CHANGELOG
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.1.7 | 2022-11-03 | #18672 | Added strict-encrypt cloud runner |
| 0.1.6 | 2022-11-01 | #18672 | Enforce to use ssl connection |
| 0.1.5 | 2022-08-05 | #15350 | Added per-stream handling |
| 0.1.4 | 2022-06-16 | #13852 | Updated stacktrace format for any trace message errors |
| 0.1.3 | 2022-05-17 | 12820 | Improved 'check' operation performance |
| 0.1.2 | 2022-02-14 | 10256 | Add -XX:+ExitOnOutOfMemoryError JVM option |
| 0.1.1 | 2022-12-05 | #9314 | Rename dynamo_db_table_name to dynamo_db_table_name_prefix. |
| 0.1.0 | 2021-08-20 | #5561 | Initial release. |