An alternative to flat file connector is to use AWS S3 connector for the source or target for your files. AWS S3 is Amazon’s object storage service. You can store the files in S3 that you would otherwise store in the hosting server. IICS provides a connector for AWS S3 that is provisioned in our IICS environments.
In order to use AWS S3 connector, you will require the following:
After you have acquired an AWS account, you can create an IAM User with Read/Write permissions to AWS S3 Buckets in your account. If you are using AWS S3 only as a source, then you may only need Read permission. An AWS Access and Secret key can be generated for this IAM User which is then used in the IICS AWS S3 Connection Properties.
IICS AWS S3 Connection Documentation
AWS S3 for testing and debugging integrations
AWS S3 provides a great mechanism to test pieces of a large ETL project. When an ETL project relies on multiple sources and transformations, small pieces of the transformation can be designed and tested by outputting to an S3 bucket for verification. Once the verification is successful, the next transformation can be designed and tested in the same fashion. This method can also be used for testing changes that need to be made on an ETL process. Parts that need to be changed can be duplicated and verified in an S3 bucket.
The WiscAlerts example describes a successful use of AWS S3 connector as a flat file connector alternative.
The Integration Platform team has implemented the integration using AWS S3 instead of relying on file system access for flat files. In this integration, the team has created an AWS S3 connection as a target location for the different files that were required for the integration. The team then created Mapping tasks that gathers information from Oracle Data sources using an Oracle connector and puts it in AWS S3 as a csv file.
The WiscAlerts example linked above describes the integration in finer detail and outlines the benefits and gaps of using AWS S3.