IICS best practices

Suggestions on how to standardize your experience and produce the best results with Informatica Intelligent Cloud Services (IICS)


Naming conventions for artifacts

These conventions can help you:

  • Locate assets quickly when searching for them in IICS web designer.
  • Provide an indication what the asset function is based on asset’s name.
  • Increase the readability when using IICS web designer.

This is an accordion element with a series of buttons that open and close related content panels.

Secure agent groups

  • Use logical names for groups
  • Use uppercase, and use underscore to separate words
  • Use documented abbreviations
  • Name structure: <group_logical_name>

Examples:

  • ODMAS
  • DCS
  • BATCH_PROCESSING_AGENT_GROUP

Secure agent

  • Name Secure Agents using logical names rather than host names
  • This way agents can be migrated easily to a different host
  • Use lowercase, and use underscore to separate words
  • Name structure: <group>_<agent_logical_name>

Examples:

  • ODMAS_test_agent
  • DCS_prod_agent
  • batch_processing_agent

Connections

  • Use documented abbreviations and use upper case for abbreviations
  • Use logical names
  • Use underscore to separate words
  • Generic name structure: <ConnectionType>_<logical_name>_<flag(s)>
    • Database: <db_type>_<logical_name>_<schema_name>
    • Flat File: FF_<logical_name>_<folder_name>

Examples:

  • ORCL_siebel_contact_center
  • FF_salesforce_lookups
  • ODBC_Accounting

Tasks

  • Use underscore to separate words
  • Include task operation in the name.
  • Use consistent naming for different type of tasks
  • Name structure: DSS_<SOURCE>_<TARGET>_<NAME>_(INSERT|UPDATE|DELETE)_<VERSION>

Examples:

  • DSS_Salesforce_Workday_Accounts_UPDATE_01
  • DSS_Salesforce_Workday_Accounts_UPDATE_02
  • DSS_Coupa_SAP_Requisition_NA_INSERT

Mappings

  • Use underscore to separate words
  • Use consistent naming for different type of mappings
  • Name structure:
    • Mapping with single source: M_<SOURCE>_<TARGET>_<NAME>_<VERSION>
    • Mapping with multiple sources: M_<SOURCE>_<ENTITY>_to_<TARGET>_<NAME>_<VERSION>

Examples:

  • M_CDI_Accounts_FF_incremental
  • M_CDI_Accounts_Contacts_to_SFDC

Task flows

  • Use underscore to separate words
  • Use consistent and readable(mix case) naming
  • Name structure: TF_<NAME>_<VERSION>

Examples:

  • TF_CDI_Acount_Contacts_to_SFDC
  • TF_Hierarchy_Change_Processing

Scheduler

  • Use underscore to separate words
  • Use consistent and readable(mix case) naming
  • Name structure: Schedule_<TIMEZONE>_<Frequency>_<TIME>

Examples:

  • Schedule_Five_Minutes
  • Schedule_PST_Daily_11PM

Project/folders

Examples:

  • DoIT-AIS-Enterprise-Integration – Salesforce Integrations

Flat file alternatives

Both AWS S3 and Box are excellent options for when you do not have easy access to the Secure Agent filesystem.

This is an accordion element with a series of buttons that open and close related content panels.

Issues with flat file connector

The IICS flat file connector requires that you have file system access on the server that hosts the IICS Secure Agent. The support overhead of managing user accounts in the hosting server will present challenges as the number of users who would require use of the connector increases. This can potentially introduce delays in project completion and provisioning/deprovisioning tasks.

Flat file connection documentation

AWS S3 as an alternative

An alternative to flat file connector is to use AWS S3 connector for the source or target for your files. AWS S3 is Amazon’s object storage service. You can store the files in S3 that you would otherwise store in the hosting server. IICS provides a connector for AWS S3 that is provisioned in our IICS environments.

In order to use AWS S3 connector, you will require the following:

After you have acquired an AWS account, you can create an IAM User with Read/Write permissions to AWS S3 Buckets in your account. If you are using AWS S3 only as a source, then you may only need Read permission. An AWS Access and Secret key can be generated for this IAM User which is then used in the IICS AWS S3 Connection Properties.

IICS AWS S3 Connection Documentation

AWS S3 for testing and debugging integrations

AWS S3 provides a great mechanism to test pieces of a large ETL project. When an ETL project relies on multiple sources and transformations, small pieces of the transformation can be designed and tested by outputting to an S3 bucket for verification. Once the verification is successful, the next transformation can be designed and tested in the same fashion. This method can also be used for testing changes that need to be made on an ETL process. Parts that need to be changed can be duplicated and verified in an S3 bucket.

WiscAlerts example

The WiscAlerts example describes a successful use of AWS S3 connector as a flat file connector alternative.

The Integration Platform team has implemented the integration using AWS S3 instead of relying on file system access for flat files. In this integration, the team has created an AWS S3 connection as a target location for the different files that were required for the integration. The team then created Mapping tasks that gathers information from Oracle Data sources using an Oracle connector and puts it in AWS S3 as a csv file.

The WiscAlerts example linked above describes the integration in finer detail and outlines the benefits and gaps of using AWS S3.

Using Box as an alternative

Box is a service that can be used to store files in a secure manner.  IICS provides a Box connector that can be used as an alternative to using flat files. There is a tutorial, on how to set up Box connector for moving flat files around.

How to get started using Box.

When to use AWS S3 vs Box

While both AWS S3 and Box are excellent alternatives instead of relying on file system access for flat file operations, following are few things you can consider when choosing between AWS S3 vs Box.

  • Box is suitable for sensitive information (however it’s not suitable for restricted data, see Box terms of use).
  • Box has a very small learning curve compared to AWS S3 and can be set up quickly. On the other hand AWS S3 (and how it fits in AWS system) has a learning curve.
  • If you are already familiar with AWS S3 and already using it, then AWS S3 may be a better alternative.
  • If you are using other AWS services, then AWS S3 will best fit in with those.

Training and experimentation

For training and experimentation, flat files are an easy and effective way to learn IICS. When possible, you should use your local machine during training. If using a hosted Secure Agent for training purposes, then we recommend that you use IICS Box connector.

Recommendation for using flat file connector

If there are limitations to effectively using AWS S3 or Box instead of flat files, we recommend that you host a Secure Agent on a machine that you manage.

Secure agent

A Secure Agent is a Java program that runs integration tasks and enables secure communication across the firewall between our organization and IICS. More about secure agent.

This is an accordion element with a series of buttons that open and close related content panels.

Costs

Secure Agents are licensed at a per-Secure-Agent rate. Secure Agents are installed at one Secure Agent per VM/host, or one Secure Agent per Docker Container. For more information on the cost of a Secure Agent license, please review the pricing documentation.

Responsibilities

By running a Secure Agent, you are responsible for ensuring the availability of the Secure Agent program and its underlying VM/host. The Secure Agent program is upgraded automatically by Informatica, but you are responsible for managing/patching the underlying operating system.

Recommendations

The DoIT Integration Platform team has experience managing secure agents. We run our secure agent in a Docker container on Linux, hosted by Amazon Web Services (AWS). The Secure Agent Docker image is available here if you want to build and try it yourself. We use AWS Elastic Container Service (ECS) to manage the secure agent deployment environment. By allowing ECS to run the Secure Agent container, we can make sure that the Secure Agent is always running because ECS would bring up a new instance if the current instance crashed.

From our experience of running secure agents, we recommend the following when running your own secure agent:

  • Reduce file system level access to the Secure Agent host: Although access to the local file system is sometimes necessary to troubleshoot integrations, we recommend avoiding using the secure agent’s file system for integrations, where possible. For integrations that deal with flat files, using Amazon S3 along with the IICS S3 connector allows an integration to use flat files without being closely tied to the underlying file system of the Secure Agent host. By using S3 instead of the Secure Agent file system, permissions and user accounts can be managed in AWS. AWS offers more self-service and automated interfaces compared to managing user accounts accounts and permissions directly in a Secure Agent host.

Avoid maintaining state in a Secure Agent host: Related to the “Pets vs. Cattle” analogy, we recommend treating Secure Agents as ephemeral components of the overall integration architecture. Accordingly, make sure configuration files and log files are stored externally to the Secure Agent host. The DoIT Integration platform team uses AWS Elastic File System to persist configuration files. By doing this, we can destroy and replace our Secure Agent container, or underlying EC2 host, with confidence that the Secure Agent will start and operate in a consistent manner.

High availability

While a single Secure Agent is most cost effective, it does introduce a risk if the Secure Agent or underlying host were to fail. By running the Secure Agent in a container platform such as AWS Elastic Container Service (ECS), you can make sure that a single Secure Agent is always running. If the Secure Agent were to crash, ECS can automatically start a new container.

This containerized single Secure Agent architecture is appropriate for scheduled ETL jobs, but event-driven integrations, such as integrations built on Cloud Application Integration (CAI), have different requirements. For event-driven integrations, we recommend running at least two Secure Agents in parallel. If one Secure Agent were to crash, the other would be able to handle requests while a new Secure Agent is brought up.

For more information on Secure Agents with Cloud Application Integration, please see this documentation from Informatica.

Secure agent for training and learning purposes

You have multiple options if you are looking for a Secure Agent for training and learning purposes.

  • Install a Secure Agent in your local computer for training and learning proposes. See installation guide for installing Secure Agent locally.
  • Use Docker image maintained by EI for installing Secure Agent locally. See documentation (mentioned under Recommendations above as well).

Test Secure Agent instance deployed by EI on AWS. See user guide for details.

Working in a shared environment

UW-Madison operates under a single shared IICS org for each environment (test and production). This means that you might see other objects in IICS, such as projects, connections, and schedules, that belong to other areas of the university.

This is an accordion element with a series of buttons that open and close related content panels.

Use User Groups to control access

When you create a new project, make sure you edit the permissions to limit who can view and change the assets within the project. If no permissions are configured on a project or the assets within it, anybody who has access to IICS will be able to view, change, run, and delete the contents of the project. Use the group that was sent when you were granted access to IICS to control who can access the contents of the project.

  1. Right-click on a project folder and click “Permissions…”
  2. Select “Add” to add bring up the group selection menu.
  3. Select the group for your team and click “Add”.
  4. Select the check boxes for Read, Update, Delete, Execute, and Change Permission.
  5. Select Save.

Namespace projects with your group name

Even though you might not have any permissions on a project, you are still able to see that the project exists when exploring all projects. To help organize the constantly growing list of projects, we recommend that you to prefix your group name before the name of the project: {group name} – {project name}

Example: “DoIT-AIS-Enterprise-Integration – Salesforce Integrations”

This structure allows you to search your group name in the project explorer to only see a list of your group’s projects. You can create subfolders within a project if further organization/hierarchy is needed.

Your group name was sent when you were granted access to IICS and you can find the name of your group by editing the permissions for a project.