Suggestions on how to standardize your experience and produce the best results with Informatica Intelligent Cloud Services (IICS)
Naming conventions for artifacts
These conventions can help you:
- Locate assets quickly when searching for them in IICS web designer.
- Provide an indication what the asset function is based on asset’s name.
- Increase the readability when using IICS web designer.
This is an accordion element with a series of buttons that open and close related content panels.
Secure agent groups
- Use logical names for groups
- Use uppercase, and use underscore to separate words
- Use documented abbreviations
- Name structure: <group_logical_name>
Examples:
- ODMAS
- DCS
- BATCH_PROCESSING_AGENT_GROUP
Secure agent
- Name Secure Agents using logical names rather than host names
- This way agents can be migrated easily to a different host
- Use lowercase, and use underscore to separate words
- Name structure: <group>_<agent_logical_name>
Examples:
- ODMAS_test_agent
- DCS_prod_agent
- batch_processing_agent
Connections
- Use documented abbreviations and use upper case for abbreviations
- Use logical names
- Use underscore to separate words
- Generic name structure: <ConnectionType>_<logical_name>_<flag(s)>
- Database: <db_type>_<logical_name>_<schema_name>
- Flat File: FF_<logical_name>_<folder_name>
Examples:
- ORCL_siebel_contact_center
- FF_salesforce_lookups
- ODBC_Accounting
Tasks
- Use underscore to separate words
- Include task operation in the name.
- Use consistent naming for different type of tasks
- Name structure: DSS_<SOURCE>_<TARGET>_<NAME>_(INSERT|UPDATE|DELETE)_<VERSION>
Examples:
- DSS_Salesforce_Workday_Accounts_UPDATE_01
- DSS_Salesforce_Workday_Accounts_UPDATE_02
- DSS_Coupa_SAP_Requisition_NA_INSERT
Mappings
- Use underscore to separate words
- Use consistent naming for different type of mappings
- Name structure:
- Mapping with single source: M_<SOURCE>_<TARGET>_<NAME>_<VERSION>
- Mapping with multiple sources: M_<SOURCE>_<ENTITY>_to_<TARGET>_<NAME>_<VERSION>
Examples:
- M_CDI_Accounts_FF_incremental
- M_CDI_Accounts_Contacts_to_SFDC
Task flows
- Use underscore to separate words
- Use consistent and readable(mix case) naming
- Name structure: TF_<NAME>_<VERSION>
Examples:
- TF_CDI_Acount_Contacts_to_SFDC
- TF_Hierarchy_Change_Processing
Scheduler
- Use underscore to separate words
- Use consistent and readable(mix case) naming
- Name structure: Schedule_<TIMEZONE>_<Frequency>_<TIME>
Examples:
- Schedule_Five_Minutes
- Schedule_PST_Daily_11PM
Project/folders
- See: Projects Namespace
- Name structure: {group name} – {project name}
Examples:
- DoIT-AIS-Enterprise-Integration – Salesforce Integrations
Flat file alternatives
Both AWS S3 and Box are excellent options for when you do not have easy access to the Secure Agent filesystem.
This is an accordion element with a series of buttons that open and close related content panels.
Issues with flat file connector
The IICS flat file connector requires that you have file system access on the server that hosts the IICS Secure Agent. The support overhead of managing user accounts in the hosting server will present challenges as the number of users who would require use of the connector increases. This can potentially introduce delays in project completion and provisioning/deprovisioning tasks.
AWS S3 as an alternative
An alternative to flat file connector is to use AWS S3 connector for the source or target for your files. AWS S3 is Amazon’s object storage service. You can store the files in S3 that you would otherwise store in the hosting server. IICS provides a connector for AWS S3 that is provisioned in our IICS environments.
In order to use AWS S3 connector, you will require the following:
- An AWS account for your team. UW Cloud Services (Navigate to AWS Onboarding Survey to get started.)
- Knowledge of AWS S3 buckets. AWS S3 Getting Started
- Knowledge of AWS IAM Users. AWS IAM Users
- Knowledge of AWS Access Keys. AWS Access Keys
After you have acquired an AWS account, you can create an IAM User with Read/Write permissions to AWS S3 Buckets in your account. If you are using AWS S3 only as a source, then you may only need Read permission. An AWS Access and Secret key can be generated for this IAM User which is then used in the IICS AWS S3 Connection Properties.
IICS AWS S3 Connection Documentation
AWS S3 for testing and debugging integrations
AWS S3 provides a great mechanism to test pieces of a large ETL project. When an ETL project relies on multiple sources and transformations, small pieces of the transformation can be designed and tested by outputting to an S3 bucket for verification. Once the verification is successful, the next transformation can be designed and tested in the same fashion. This method can also be used for testing changes that need to be made on an ETL process. Parts that need to be changed can be duplicated and verified in an S3 bucket.
WiscAlerts example
The WiscAlerts example describes a successful use of AWS S3 connector as a flat file connector alternative.
The Integration Platform team has implemented the integration using AWS S3 instead of relying on file system access for flat files. In this integration, the team has created an AWS S3 connection as a target location for the different files that were required for the integration. The team then created Mapping tasks that gathers information from Oracle Data sources using an Oracle connector and puts it in AWS S3 as a csv file.
The WiscAlerts example linked above describes the integration in finer detail and outlines the benefits and gaps of using AWS S3.
Using Box as an alternative
Box is a service that can be used to store files in a secure manner. IICS provides a Box connector that can be used as an alternative to using flat files. There is a tutorial, on how to set up Box connector for moving flat files around.
How to get started using Box.
When to use AWS S3 vs Box
While both AWS S3 and Box are excellent alternatives instead of relying on file system access for flat file operations, following are few things you can consider when choosing between AWS S3 vs Box.
- Box is suitable for sensitive information (however it’s not suitable for restricted data, see Box terms of use).
- Box has a very small learning curve compared to AWS S3 and can be set up quickly. On the other hand AWS S3 (and how it fits in AWS system) has a learning curve.
- If you are already familiar with AWS S3 and already using it, then AWS S3 may be a better alternative.
- If you are using other AWS services, then AWS S3 will best fit in with those.
Training and experimentation
For training and experimentation, flat files are an easy and effective way to learn IICS. When possible, you should use your local machine during training. If using a hosted Secure Agent for training purposes, then we recommend that you use IICS Box connector.
Recommendation for using flat file connector
If there are limitations to effectively using AWS S3 or Box instead of flat files, we recommend that you host a Secure Agent on a machine that you manage.
Secure agent
A Secure Agent is a Java program that runs integration tasks and enables secure communication across the firewall between our organization and IICS. More about secure agent.
This is an accordion element with a series of buttons that open and close related content panels.
Costs
Secure Agents are licensed at a per-Secure-Agent rate. Secure Agents are installed at one Secure Agent per VM/host, or one Secure Agent per Docker Container. For more information on the cost of a Secure Agent license, please review the pricing documentation.
Responsibilities
By running a Secure Agent, you are responsible for ensuring the availability of the Secure Agent program and its underlying VM/host. The Secure Agent program is upgraded automatically by Informatica, but you are responsible for managing/patching the underlying operating system.
Recommendations
The DoIT Integration Platform team has experience managing secure agents. We run our secure agent in a Docker container on Linux, hosted by Amazon Web Services (AWS). The Secure Agent Docker image is available here if you want to build and try it yourself. We use AWS Elastic Container Service (ECS) to manage the secure agent deployment environment. By allowing ECS to run the Secure Agent container, we can make sure that the Secure Agent is always running because ECS would bring up a new instance if the current instance crashed.
From our experience of running secure agents, we recommend the following when running your own secure agent:
- Reduce file system level access to the Secure Agent host: Although access to the local file system is sometimes necessary to troubleshoot integrations, we recommend avoiding using the secure agent’s file system for integrations, where possible. For integrations that deal with flat files, using Amazon S3 along with the IICS S3 connector allows an integration to use flat files without being closely tied to the underlying file system of the Secure Agent host. By using S3 instead of the Secure Agent file system, permissions and user accounts can be managed in AWS. AWS offers more self-service and automated interfaces compared to managing user accounts accounts and permissions directly in a Secure Agent host.
Avoid maintaining state in a Secure Agent host: Related to the “Pets vs. Cattle” analogy, we recommend treating Secure Agents as ephemeral components of the overall integration architecture. Accordingly, make sure configuration files and log files are stored externally to the Secure Agent host. The DoIT Integration platform team uses AWS Elastic File System to persist configuration files. By doing this, we can destroy and replace our Secure Agent container, or underlying EC2 host, with confidence that the Secure Agent will start and operate in a consistent manner.
High availability
While a single Secure Agent is most cost effective, it does introduce a risk if the Secure Agent or underlying host were to fail. By running the Secure Agent in a container platform such as AWS Elastic Container Service (ECS), you can make sure that a single Secure Agent is always running. If the Secure Agent were to crash, ECS can automatically start a new container.
This containerized single Secure Agent architecture is appropriate for scheduled ETL jobs, but event-driven integrations, such as integrations built on Cloud Application Integration (CAI), have different requirements. For event-driven integrations, we recommend running at least two Secure Agents in parallel. If one Secure Agent were to crash, the other would be able to handle requests while a new Secure Agent is brought up.
For more information on Secure Agents with Cloud Application Integration, please see this documentation from Informatica.
Secure agent for training and learning purposes
You have multiple options if you are looking for a Secure Agent for training and learning purposes.
- Install a Secure Agent in your local computer for training and learning proposes. See installation guide for installing Secure Agent locally.
- Use Docker image maintained by EI for installing Secure Agent locally. See documentation (mentioned under Recommendations above as well).
Test Secure Agent instance deployed by EI on AWS. See user guide for details.