The term “data pipeline” refers to a collection of processes that gather raw data and convert it into an format that can be utilized by software programs. Pipelines can be batch or real-time. They can be installed on-premises or in the cloud, and their tooling can be commercial or open source.
Similar to how a pipeline transports water from the river to your home, data pipelines bring data from one layer (transactional or event sources) to another (data lakes and warehouses). This allows analytics and insights to be extracted from the data. In the past transfer of this data required manual procedures like daily uploads and long wait times for insights. Data pipelines replace manual processes click for source and enable companies to transfer data more efficiently and without risk.
Accelerate development using a virtual pipeline of data
A virtual data pipe can save lots of money on infrastructure costs such as storage in the datacenter or in remote offices. It also reduces hardware, network and administration costs for non-production environments like test environments. It can also save time due to automation of data refresh, masking, role based access control, and database customization and integration.
IBM InfoSphere Virtual Data Pipeline (VDP) is an all-cloud copy management solution that decouples test and development environments from production infrastructures. It uses patented snapshot and changed-block tracking technology to capture application-consistent copies of databases and other files. Users can mount masked, near-instant virtual copies of databases in non-production environments, and begin testing in just minutes. This is particularly beneficial for accelerating DevOps, agile methodologies and speeding time to market.
Lascia un commento