Build And Run Continuous Ingestion Flows By Streaming Ingestion Feature Of Data Ingestion
IICS Streaming Ingestion services bring streaming capabilities into Informatica Intelligent Cloud Services (IICS). Data Ingestion allows to ingest data along with real time streaming data at a scale from source to targets with low latency. This is the new feature provided by Informatica Cloud services and is available in ‘My Services’ page which has the list of services available associated with secure agent. Streamed data can be obtained from Facebook, Change Data Capture (CDC), logs, Twitter or any real-time data which can be of high volume and can be integrated to on-premises for further analysis and improvements.
To make use of streaming ingestion task, create, deploy and monitor the task with the help of these steps:
- In the Administration page of Informatica Cloud Services, Streaming Ingestion Agent Service is listed under System Configuration details. Configure the details of Engine, Agent and Agent Log.
- Configure the source connections pointing to streaming source and target to which the data needs to be loaded.
- Configure and create the streaming ingestion task.
- Deploy the streaming ingestion task.
- Monitor the streaming ingestion task and jobs.
Communication Architecture of Streaming Ingestion Agent
Transformations available in Streaming Ingestion Task:
- Filter – Filters out bad or erroneous records from the data we are ingesting
- Segregator – Segregates data from source onto multiple target partitions
- Combiner – Combines multiple messages into single file before data is ingested to target
Once the data flow for streaming ingestion task is developed and validated, it can be deployed and monitored. Deployed job has the status of “Up and Running” with a pictorial representation of job and details of throughput on the system.
Each job has monitoring details of the data flow that is developed with details of:
- Overview – Gives details of Job, Duration, Runtime environment, State, Started by
- Alert – Displays alert message when an event occurs
- Performance – Shows how the job has been varying with respect to average throughput
- Past Run – Gives statistics and status of streaming ingestion jobs of the previous run
Monitoring Service to monitor jobs has Data Ingestion which has Data Ingestion specific jobs that gives overview of present deployment status across all the Mass Ingestion Services.
Benefits of Streaming Ingestion Task:
- Expedite data delivery to accredit real-time analytics and data feeds
- Reduce manual efforts and skill for managing data stream processing
- Query and process over data within rolling time window or the most recent record
Streaming Ingestion Service provides benefit of capturing real time data that is generated dynamcially. It also avoids the need to wait as the data is captured as a continuous stream. It is really helpful to capture real time incremental data, thereby reducing manual efforts.
Contact for further details
Sr. Technology Specialist