Many enterprise customers are looking for data delivery solutions that can help ramp up their Data Analytics game at a much lower cost than trying to build the technologies themselves. The decision to build a solution could result in accumulation of technical debt; extensive/unpredictable time to production; potential vendor lock for maintenance of custom-made technologies designed and developed by consultants; tracking cobbled together components created by multiple staff and consultants; and much higher costs for future growth/scaling.

Fortunately, Treehouse Software brings customers TDT-DIRECT, a Cloud-native, turn-key solution for fully automated bulk load and CDC data transfer to the top Analytics/ML/AI-friendly targets, such as Amazon Redshift, Amazon Athena/S3, Snowflake, Amazon S3 Express One Zone Buckets, and Amazon Aurora PostgreSQL (all the while adhering to AWS’s and Snowflake’s recommended “best practices” for massive data loading, thus assuring shortest and surest loads). 

Additionally, TDT-DIRECT includes the infrastructure needed to autogenerate all schemas, tables, views, etc. required for loading massive quantities of data into the various targets. 

Another important item of note is that all TDT-DIRECT Lambda microservices are fully customizeable (they will be YOUR Lambdas) to add extra monitoring capabilities, and any other functionalities for future needs.

Example of how TDT-DIRECT uses Snowflake’s best practices vs. traditional ODBC…

TDT-DIRECT’s innovative Lambda-based microservices approach enables faster data flow than any conceivable ODBC-based solution, which is the standard tool used for most “roll your own” approaches, or “we have a connector for that” offerings.  

To load massive quantities of data to a target, TDT-DIRECT uses Snowflake’s (hugely scalable) bulk load utilities—not ODBC. It is vital to note that Snowflake is NOT a relational (OLTP) database, so doing CDC transfers to this target via ODBC (with update, insert, delete transactions) goes directly against “best practices” advice from Snowflake, and would almost assuredly result in unwieldy bottlenecks.

TDT-DIRECT loads data into Snowflake’s “delta tables”, which inherently retain the entire history of source data ever since the source-to-target synchronization began (perfect for time-based trend/predictive/prescriptive analytics). Again, TDT adheres to Snowflake’s best practices recommendation for pulling data from S3 for bulk loading massive quantities of data.

TDT-DIRECT leverages AWS CloudFormation for ease of implementation...

Treehouse provides highly-detailed CloudFormation Templates which automate and accelerate the process of installing and configuring the complete TDT-DIRECT application (including AWS Lambda functions and a number of other AWS resources) in your AWS account(s). The TDT-DIRECT CloudFormation Templates create stacks consisting of all principal framework components, along with related IAM policies and roles which are carefully engineered to comply with “best practices” (such as a “least privileges” approach to permissions).

The TDT-DIRECT CloudFormation Templates also optionally provide for automatic creation of a VPC, its subnets, and all required standard VPC-oriented resources, as well as optional creation of a source database cluster (consisting of either a sample database provided by Treehouse for a quick trial/POC, or your own database and data).

Simply put, TDT-DIRECT is a Cloud-native, turn-key solution that can eliminate months (or even years) of research and development time and costs, and allow customers to be up and running in minutes. With TDT-DIRECT, customers benefit from high-speed, large scale data movement that strictly adheres to AWS’s and Snowflake’s recommended use of extremely scalable bulk load utilities. This adherence to best practices is a key differentiator of TDT-DIRECT from other “connector” offerings on the market. TDT-DIRECT provides the turn-key solution for rapidly transferring data to advanced Analytics/ML/AI-friendly targets.

Cloud Engineering for Data Transfer Professional Services (Available on the AWS Marketplace): Treehouse Software's Cloud engineers provide deep ELT/ETL data transfer expertise related to the implementation, customization, optimization, and overall use of Treehouse Dataflow Toolkit (TDT) to meet specific business goals.

 

Treehouse Dataflow Toolkit (TDT) is Copyright © 2024 Treehouse Software, Inc. All rights reserved.

---

TDT for Mainframe Data Sources

Download the AWS TDT Product Brief

Request a Demo