• IBM Consulting

    DBA Consulting can help you with IBM BI and Web related work. Also IBM Linux is our portfolio.

  • Oracle Consulting

    For Oracle related consulting and Database work and support and Migration call DBA Consulting.

  • Novell/RedHat Consulting

    For all Novell Suse Linux and SAP on Suse Linux questions releated to OS and BI solutions. And offcourse also for the great RedHat products like RedHat Enterprise Server and JBoss middelware and BI on RedHat.

  • Microsoft Consulting

    For Microsoft Server 2012 onwards, Microsoft Client Windows 7 and higher, Microsoft Cloud Services (Azure,Office 365, etc.) related consulting services.

  • Citrix Consulting

    Citrix VDI in a box, Desktop Vertualizations and Citrix Netscaler security.

  • Web Development

    Web Development (Static Websites, CMS Websites (Drupal 7/8, WordPress, Joomla, Responsive Websites and Adaptive Websites).

20 April 2020

GridGain In-Memory Computing Platform Technology Overview



Introduction to Oracle Database In-Memory


Oracle Database In-Memory (Database In-Memory) is a suite of features, first introduced in Oracle Database 12c Release 1 (12.1.0.2), that greatly improves performance for real-time analytics and mixed workloads. The In-Memory Column Store (IM column store) is the key feature of Database In-Memory.

Note: Database In-Memory features require the Oracle Database In-Memory option.
We discuss the following topics:

Oracle Database In Memory – Oracle’s Dual Format Database Delivers Real Time Analytics - IMC Summit


Challenges for Analytic Applications

Traditionally, obtaining good performance for analytic queries meant satisfying several requirements.
The Single-Format Approach

Traditionally, relational databases store data in either row or columnar formats. Memory and disk store data in the same format.

The Oracle Database In-Memory Solution

The Oracle Database In-Memory (Database In-Memory) feature set includes the In-Memory Column Store (IM column store), advanced query optimizations, and availability solutions.
Prerequisites for Database In-Memory

The Oracle Database In-Memory option is required for all Database In-Memory features. No special hardware is required for an IM column store.

Principal Tasks for Database In-Memory

For queries to benefit from the IM column store, the only required tasks are sizing the IM column store, and specifying objects for population. Query optimization and availability features require additional configuration.

Tools for the IM Column Store

No special tools or utilities are required to manage the IM column store or other Database In-Memory features. Administrative tools such as SQL*Plus, SQL Developer, and Oracle Enterprise Manager (Enterprise Manager) are fully supported.




The GridGain® in-memory computing platform is deployed as a cluster of commodity servers between your Oracle database and application layer. GridGain integrates seamlessly with your Oracle databases and, through the GridGain Unified API, with your application.

GridGain® Systems, an Oracle Silver Partner, has certified the GridGain In-Memory Computing Platform to run on Oracle Cloud Infrastructure. The GridGain platform in the Oracle Cloud Marketplace is available under the BYOL (bring your own license) model.

Why we love Oracle Data Cloud Tel Aviv

In-Memory Computing Powers Digital Transformations

GridGain, built on Apache® Ignite™, delivers in-memory speed and massive scalability to data-intensive applications with no rip-and-replace of the underlying RDBMS, NoSQL or Hadoop database for existing applications. GridGain solutions are used by enterprises in financial services, fintech, software, ecommerce, retail, online business services, healthcare, telecom and other major sectors.

Often used for digital transformation initiatives, the comprehensive GridGain platform includes an in-memory data grid, in-memory database, streaming analytics, and a continuous learning framework. GridGain enables high-performance ACID transactions, real-time streaming, continuous learning and fast analytics in a single, comprehensive data access and processing layer.

Easy to Integrate into New or Existing Architectures

GridGain is easy to integrate into existing or new architectures. It includes a unified API which supports application layer access via SQL, C++, .NET, JAVA/Scala/Groovy, Node.js and more. It integrates seamlessly with RDBMS, NoSQL and Hadoop databases while offering ACID transactions and ANSI SQL-99 compliance. GridGain includes native integrations or compatibility with popular solutions including Apache® Kafka™, Apache® Spark™, Kubernetes, Redis, Memcached and many more.

The memory-centric GridGain platform architecture leverages ongoing advancements in memory and storage technologies to provide distributed in-memory computing performance with the cost and durability of disk storage. Users can achieve a 1,000x or more increase in performance while scaling out to petabytes of in-memory data across a cluster of commodity servers. GridGain can run on-premises, on a cloud platform, or in a hybrid environment.

The GridGain Enterprise Edition

The GridGain Enterprise Edition is built on Apache Ignite and adds enterprise-grade features including data center replication, enterprise-grade security, rolling upgrades, and more. The Enterprise Edition is extensively tested by GridGain Systems and is for production use in large-scale or mission-critical deployments, or environments with heightened security requirements.

Learn More About GridGain on the Oracle Cloud


The GridGain Enterprise Edition is a subscription-based product and is included as part of a GridGain Support package. Check the GridGain listing in the Oracle Cloud Marketplace for further information.

GridGain In-Memory Computing Platform Technology Overview


The GridGain® in-memory computing platform dramatically accelerates and scales out data-intensive applications across a distributed computing architecture. GridGain solves the performance needs of companies launching digital transformation, omnichannel customer experience, Internet of Things, or similar data-intensive initiatives. GridGain is built on Apache® Ignite™, which was originally contributed to the Apache Software Foundation (ASF) by GridGain Systems. Ignite has become a top five ASF project and has been downloaded millions of times since the project launched in 2014.

Distributed Caching

The comprehensive GridGain in-memory computing solution includes an in-memory data grid, in-memory database, streaming analytics, and a continuous learning framework. The system provides ANSI-99 SQL and ACID transaction support. The platform can function as an in-memory data grid or it can be deployed as a memory-centric database which combines the speed of in-memory computing with the durability of disk-based storage. Performance is 1,000x faster than systems built on traditional disk-based databases because all data is held in-memory and the system utilizes massive parallel processing. The system can be scaled out by adding new nodes to the cluster to support up to petabytes of data from multiple databases with automatic data rebalancing and redundant storage across the nodes.

GridGain can modernize existing data-intensive architectures when inserted between existing application and data layers. GridGain integrates seamlessly with RDBMS, NoSQL and Hadoop databases and includes a unified API which supports SQL, C++, .NET, JAVA/Scala/Groovy, and Node.js access for the application layer. It integrates with many common management and data processing solutions through an ODBC/JDBC API. GridGain can run on-premises, on a private or public cloud platform such as AWS, Microsoft Azure or Google Cloud Platform, or on a hybrid environment.

Future of Data Integration: Data Mesh, with a Deep Dive into GoldenGate, Kafka and Spark




Deployments

GridGain Software Leverages the Ongoing Developments of the Apache Ignite Project

The GridGain Community Edition, built on Apache Ignite, includes patches and updates not yet released in Ignite with additional features and QA testing to provide the highest possible performance, reliability and managability possible in an open source in-memory computing platform.

The GridGain Enterprise Edition is for organizations using GridGain in-memory data grid in mission-critical applications, and includes additional security, monitoring, and management features that are not available in the GridGain Community Edition. The Enterprise Edition is hardened and undergoes extensive testing to ensure its performance in mission-critical production environments where performance, reliability and high availability are critical to success.

The GridGain Ultimate Edition is for organizations using GridGain as an in-memory database in mission-critical applications and includes all of the features in the Enterprise Edition, plus highly recommended backup and recovery features.



Introduction to Apache Ignite


Learn about the key capabilities and features of the Apache® Ignite™ in-memory computing platform and how it adds speed and scalability to existing and new applications. Download this free white paper entitled "Introduction to Apache Ignite" for a deep dive into the Apache Ignite architecture, APIs, features, and use cases.

Matt Coventon - Test Driving Streaming and CEP on Apache Ignite

APACHE IGNITE OVERVIEW

Companies pursuing digital transformation and omnichannel customer experience initiatives often find that their existing IT infrastructure is unable to support the speed and scalability these initiatives demand. The need to provide a more personalized, real-time end-user experience has forced companies to transform batch-based processes that take days or hours into real-time, automated processes that take seconds or less. Companies that have adopted web, mobile, social or IoT technologies to support their transformation often experience 10-1000x increases in queries and transactions. These initiatives typically create a host of new data sources about the end users and business. This is reflected in a worldwide 50x explosion in data volumes over the last decade.

How can companies support 10-1000x increases in query and transaction volumes, leverage 50x as much data for decision making, and do everything that used to take hours or days in seconds or fractions of a second? The answer for many companies has been in-memory computing. In-memory computing offers speed and scalability for existing and new applications. The speed comes from storing and processing data in memory rather than continually retrieving data from disk before processing. While hard drive (HDD) media speeds are measured in milliseconds, RAM speeds can be measured in nanoseconds, a million times faster. Scalability comes from distributing data and computing together across a cluster of servers.

Introducing Oracle Velocity Scale, a high performance, scale out, shared nothing SQL In Memory RDBMS


GRIDGAIN APACHE IGNITE TUTORIAL VIDEOS

For more information on Apache Ignite and using Apache Ignite in production, refer to the list of video presentations and webinars below:


  • Getting Started with Apache Ignite as a Distributed Database
  • Introducing Apache Ignite (Part 1)
  • Introduction Apache Ignite (Part 2)
  • Moving Apache Ignite Into Production

THE APACHE IGNITE IN-MEMORY COMPUTING PLATFORM

Apache Ignite (Ignite) is the leading Apache Software Foundation (ASF) project for in-memory computing. It is one of the top five ASF projects in terms of commits and email list activity. Ignite is an in-memory computing platform that includes an in-memory data grid (IMDG), in-memory database (IMDB), support for streaming analytics, and a continuous learning framework for machine and deep learning. It provides in-memory speed and unlimited horizontal scalability to:

Existing or new OLTP or OLAP applications

New or existing hybrid transactional/analytical processing (HTAP) applications
Support for streaming analytics Continuous learning use cases involving machine or deep learning
The source code for Apache Ignite was originally contributed to the Apache Software Foundation by GridGain Systems. The Apache Ignite middleware project rapidly evolved into a top-level Apache Software Foundation project and now has generated millions of downloads since its inception in 2014.

APACHE IGNITE BENEFITS AND FEATURES

Apache Ignite includes the following powerful in-memory computing platform benefits and features:


  • An In-Memory Data Grid
  • An In-Memory Database
  • A Streaming Analytics Engine
  • A Continuous Learning Framework for machine and deep learning
  • A Persistent Store
  • An In-Memory Compute Grid
  • An In-Memory Service Grid
  • Advanced Clustering
  • An Accelerator for Hadoop
  • An In-Memory Distributed File System

And much more.

TimesTen Scaleout: Functionality, Architecture and Performance Tuning

APACHE IGNITE ARCHITECTURE

MEMORY-CENTRIC STORAGE

Ignite provides a distributed in-memory data store that delivers in-memory speed and unlimited read and write scalability to applications. It is a distributed, in-memory SQL and key-value store that supports any kind of structured, semi-structured and unstructured data. Ignite’s unlimited horizontal scalability comes from a shared-nothing, node-based cluster architecture. Each node delivers low latency and predictable access times using an in-memory first architecture that stores data in off-heap RAM by default.

THIRD PARTY PERSISTENCE AND THE IGNITE NATIVE PERSISTENCE

Ignite can bring together almost any data into memory and deliver unlimited read scalability on top of third-party databases. Ignite can sit as an in-memory data grid (IMDG) on top of all popular RDBMSs such as IBM DB2®, Microsoft SQL Server®, MySQL®, Oracle®, and PostgreSQL®. It also works with NoSQL databases such as Apache Cassandra™ or MongoDB® and with Apache Hadoop™. Ignite also provides its own native persistence, a distributed in-memory database (IMDB). Its performance for high volume, low latency transactions and data ingestion exceeds the read and write performance and scalability of traditional databases. Ignite can sit on top of all these databases at the same time as an IMDG and coordinate transactions in-memory with the underlying databases to ensure data is never lost.

ANSI-99 COMPLIANT SQL

Unlike other in-memory data grid technologies, Ignite supports high performance, low latency ANSI-99 compliant distributed SQL. Ignite is the only vendor that supports SQL DDL and DML for real-time or batch queries and transactions, for any data spread across any combination of third-party databases or Ignite’s native persistence. This enables companies to use their existing SQL assets and SQL skillsets with in-memory computing instead of having to rewrite applications or replace databases.

SUPPORT FOR ACID TRANSACTIONS

Ignite also has the broadest support for distributed ACID transactions. Ignite’s integrated SQL and ACID transaction support makes it the only technology that can add speed and scalability by sliding in-between SQL-based applications and RDBMSs and preserving the use of SQL. Ignite intercepts all SQL queries and transactions. This architecture has offloaded all queries from existing databases, lowered SQL query times 10-1000x and delivered unlimited horizontal read scalability. It has given companies a way to handle the increased loads without having to rip out and replace existing applications and databases. It has also provided a way to easily migrate data at any time from existing databases to Ignite’s native persistence as an IMDB to improve transaction and write performance.

MASSIVELY PARALLEL PROCESSING (MPP)

Ignite also eliminates another common bottleneck associated with Big Data: the network. Data has gotten so big  that just moving it takes minutes or hours. With Ignite, data can be distributed to nodes based on data affinity declarations that help collocate processing with data. This improves performance and scalability by reducing data transmission over the network. Ignite provides the broadest implementation of distributed MPP algorithms to enable collocated processing. It supports the parallelization of distributed computations based on Fork/Join and extensively uses distributed parallel computations internally. Ignite’s distributed SQL and key- value APIs, MapReduce, compute grid, service grid, streaming analytics and machine learning capabilities all leverage MPP. Ignite also exposes APIs so developers can add and distribute their own user-defined MPP functionality using C++, Java, or .NET.

The combination of Ignite’s shared-nothing architecture and MPP delivers unlimited horizontal, linear scalability. Companies can scale out data and MPP horizontally using any combination of servers or cloud infrastructure. This approach pays for itself by providing a much more cost-effective alternative than scaling existing databases or applications that can often only be scaled vertically with expensive hardware.

IN-MEMORY SUPPORT FOR THIRD-PARTY TECHNOLOGIES

Ignite enables in-memory computing to be used for just about any type of project by providing the broadest third-party technology support. This includes support for all leading RDBMSs, for NoSQL databases including Cassandra and MongoDB, and the broadest integration with Apache Spark™ and Apache Hadoop. When used with these technologies, Ignite transparently adds in-memory access to any data with the speed, scalability, and power of Ignite.

Check out Apache Ignite vs Apache Spark: Integration using Ignite RDDs for information on using Apache Ignite and Apache Spark together.

UNIFIED API

Ignite makes data accessible anywhere as SQL or key-values through a unified API. You can securely access any data in any cluster regardless of its deployment configuration. It is implemented across:


  • SQL (ODBC and JDBC)
  • REST, .NET, C++, Java/Scala/Groovy and SSL/TLS
  • A binary protocol that supports lightweight clients for any language
  • LINEAR HORIZONTAL SCALABILITY


Ignite is a Java-based in-memory computing platform. It provides one of the most sophisticated horizontal scale-out and clustering architectures available. The architecture delivers elastic scalability: nodes can be dynamically added to, removed from, or upgraded in a running cluster with zero downtime. Nodes can automatically discover each other across LANs, WANs and any cloud infrastructure.

All nodes in an Ignite cluster are equal and can play any logical role. The node flexibility is achieved by using a Service Provider Interface (SPI) design for every internal component. The SPI model makes Ignite fully customizable and pluggable. This not only enables tremendous configurability of the system. It gives companies the flexibility to support existing and future server infrastructure on-premises or in the cloud.

DEPLOYABLE ANYWHERE

Ignite is easy to use and deploy anywhere on commodity servers. It can be deployed on-premises, in a private cloud using Kubernetes®, Docker® or other technologies, on public cloud platforms such as Amazon Web Services (AWS®), Microsoft Azure® or Google Cloud Platform®, or on a hybrid private-public cloud.

S3D Valentin How to Add Speed and Scale to SQL, Support New Data Needs, and Keep Your RDBMS




APACHE IGNITE USE CASES

Below is a general Apache Ignite tutorial on how and why companies deploy Apache Ignite in production.

Ignite is typically used to:


  • Add speed and scalability to existing applications
  • Build new, modern, highly performant and scalable trans- actional and/or analytical applications
  • Build streaming analytics applications, often with Apache Spark, Apache Kafka™ and other streaming technologies
  • Add continuous machine and deep learning to applications to improve decision automation

Companies start in any one of these areas. Over time, as Ignite is used with more projects, it becomes a common in-memory data access layer that can support the data, performance and scalability needs for any new workload:

Data services and other APIs that help deliver an omni- channel digital business
New customer-facing applications that support new products and services
Real-time analytics that help improve operational visibility or compliance
Streaming analytics, machine and deep learning that help improve the customer experience and business outcomes Ignite helps deliver these types of projects faster while giving companies a foundation for a more real-time, responsive digital business model and the ability to be more flexible to change.

IN-MEMORY DATA GRID (IMDG) FOR ADDING SPEED AND SCALABILITY TO APPLICATIONS

A core Ignite capability and most common use case is as an IMDG. Ignite can increase the performance and scalability  of existing applications and databases by sliding in-between the application and data layer with no rip-and-replace of the database or application and no major architectural changes. Ignite supports all common RDBMSs including IBM DB2, Microsoft SQL Server, MySQL, Oracle and PostgreSQL, NoSQL databases such as Cassandra and MongoDB, and Hadoop.

Ignite generates the application domain model based on the schema definition of the underlying database. It then loads the data and acts as the data platform for the application. Ignite handles all reads and coordinates transactions with the underlying database in a way that ensures data consistency in both the database and Ignite. By utilizing RAM in place of disk, Ignite lowers latency by orders of magnitude compared to traditional disk-based databases.

In-Memory Data Grid diagram
The primary benefits and capabilities of the Ignite IMDG include:


  • ANSI SQL-99 support including DML and DDL
  • ACID transaction support
  • In-memory performance orders of magnitude faster than disk-based RDBMSs
  • Distributed in-memory caching that offloads queries from the existing database
  • Elastic scalability to handle up to petabytes of in-memory data
  • Distributed in-memory queue and other data structures
  • Web session clustering
  • Hibernate L2 cache integration
  • Tiered off-heap storage
  • Deadlock-free transactions for fast in-memory transaction processing
  • JCache (JSR 107), Memcached and Redis client APIs that simplify migration from existing caches
HYBRID IN-MEMORY DATABASE (IMDB) FOR HIGH VOLUME, LOW LATENCY TRANSACTIONS AND DATA INGESTION

An Ignite cluster can also be used as a distributed, transactional IMDB to support high volume, low latency transactions, and data ingestion, or for low-cost storage.

The Ignite IMDB combines distributed, horizontally scalable ANSI-99 SQL and ACID transactions with Ignite’s native persistence. It supports all SQL, DDL and DML commands including SELECT, UPDATE, INSERT, MERGE and DELETE queries

and CREATE and DROP table. Ignite parallelizes commands whenever possible, such as distributed SQL joins. It allows for cross-cache joins across the entire cluster, which includes joins between data persisted in third-party databases and Ignite’s native persistence. It also allows companies to put 0-100% of data in RAM for the best combination of performance and cost.

In-Memory Database diagram
The in-memory distributed SQL capabilities allow developers, administrators and analysts to interact with the Ignite platform using standard SQL commands through JDBC or ODBC or natively developed APIs across other languages as well.

The primary capabilities of the Ignite’s hybrid IMDB include:


  • ANSI SQL-99 compliance
  • ACID transactions support
  • Full support for SQL DML including SELECT, UPDATE, IN- SERT, MERGE and DELETE
  • Support for DDL commands including CREATE and DROP table
  • Support for distributed SQL joins, including cross-cache joins across the entire cluster
  • SQL support through JDBC and ODBC without custom coding
  • Geospatial support
  • Hybrid memory support for RAM, HDD, SSD/Flash, 3D XPoint and other storage technologies
  • Support for maintaining 0-100% of data in RAM with the full data set in non-volatile storage
  • Immediate availability on restart without having to wait for RAM warm-up


Autonomous Data Management: The Next Generation Data Platform - IMC Summit North America 2019



STREAM INGESTION, DATA MANAGEMENT, PROCESSING AND REAL-TIME ANALYTICS FOR STREAMING ANALYTICS

Ignite is used by the largest companies in the world to ingest, process, store and publish streaming data for large-scale, mission-critical business applications. It is used by several of the largest banks in the world for trade processing, settlement, and compliance; by telecommunications companies to deliver call services over telephone networks and the Internet; by retailers and e-commerce vendors to deliver an improved real-time experience; and by leading cloud infrastructure and SaaS vendors as the in-memory computing foundation of their offerings. Companies have been able to ingest and process streams with millions of events per second on a moderately-sized cluster.

Streaming Analytics diagram
Ignite is integrated and used with major streaming technologies including Apache Camel™, Kafka, Spark, and Storm™, Java Message Service (JMS) and MQTT to ingest, process and publish streaming data. Once loaded into the cluster, companies can leverage Ignite’s native MPP-style libraries for concurrent data processing, including concurrent SQL queries and continuous learning. Clients can then subscribe to continuous queries which execute and identify important events as streams are processed.

Ignite also provides the broadest in-memory computing integration with Apache Spark. The integration includes native support for Spark DataFrames, an Ignite RDD API for reading in and writing data to Ignite as mutable Spark RDDs, optimized SQL, and an in-memory implementation of HDFS with the Ignite File System (IGFS). When deployed together, Spark can access all of the in-memory data in Ignite, not just data streams; share data and state across all Spark jobs; and take advantage of all of Ignite’s in-memory loading and processing capabilities including continuous learning to train models in near real-time to improve outcomes for in-process HTAP applications.

CONTINUOUS LEARNING FRAMEWORK FOR MACHINE LEARNING AND DEEP LEARNING

Continuous Learning Framework diagram
Ignite also provides Ignite Machine Learning (ML), MPP-style machine learning and deep learning with real-time performance on petabytes of data. Ignite provides several standard machine learning algorithms optimized for collocated processing including linear and multi-linear regression, k-means clustering, decision trees, k-NN classification and regression. It also includes a multilayer perceptron for deep learning along with TensorFlow integration. Developers can develop and deploy their own algorithms across any cluster as well as using the compute grid.

CORE FEATURES OF APACHE IGNITE

Ignite combines a core set of capabilities that support many use cases in one integrated platform. It includes a high-performance in-memory data store combined with support for both third-party persistence and native persistence. It offers a broad range of MPP capabilities that include distributed SQL, a general-purpose compute grid, a service grid for micro-services, stream processing, and analytics, and machine and deep learning. It provides integration with third-party technologies to add in-memory computing to a host of use cases. It also exposes a unified API across languages that enables nearly any type of client to access these capabilities.

IN-MEMORY DATA STORE

Ignite provides an in-memory data store where each node in an Ignite cluster by default stores data in RAM. The data is kept in off-heap storage to ensure low latency and consistent access times. The system is multi-model, with the ability to support structured, semi-structured and unstructured data. The data is accessible via SQL using JDBC/ODBC drivers or APIs, or using key-value APIs. Ignite provides a durable memory architecture where in-memory data can be mapped to leading third-party databases or to Ignite’s native persistence to save data within the same cluster. This sup- port enables companies to bring almost any data into RAM and create a single in-memory data layer across existing and new systems.

THIRD-PARTY DATABASE AND NATIVE PERSISTENCE SUPPORT

Ignite can be deployed as an IMDG on top of all leading relational databases (including IBM DB2, Microsoft SQL Server, MySQL, Oracle, and PostgreSQL), NoSQL databases (such as Cassandra or MongoDB), and Hadoop and use these databases for persistence. When used with a third-party database, Ignite holds the most up-to-date version of the data in-memory. Whenever a transaction occurs, Ignite passes it to the underlying database and, upon successful completion, updates the data in-memory. This enables Ignite to offload all reads from databases, increasing speed and providing existing systems a lot more room for growth.

Ignite also offers native persistence that allows Ignite to be used as a distributed hybrid IMDB. It is a multi-model database that combines ANSI-99 compliant SQL with key-value API and ACID transaction support. It is also a hybrid memory database that supports any combination of RAM, HDD, SSD/ Flash, 3D XPoint and other storage technologies.

Ignite’s native persistence lets you choose the best combination of performance and cost for each situation. With Ignite’s native persistence you can have 0-100% of your data and SQL indexes in RAM across the cluster while the full data set and indexes are stored in non-volatile storage to provide durability and availability. All data is stored and treated the same way. Ignite uses write-ahead logs for transactions and incremental snapshots optimized to ensure in-memory speed, data consistency and recoverability.

Ignite’s native persistence also provides immediate recovery in the case of a failure. You do not need to wait for data  to be loaded in RAM after a cluster restart. All operations use data stored on disk until that data is loaded into RAM. Immediate recovery is one of the many features that have helped companies ensure high availability.

Driving efficient mainframe digital transformation leveraging GridGain Ignite on z OS


COMPUTE GRID

Ignite provides a compute grid which enables parallel, in-memory processing of CPU-intensive or other resource-intensive tasks. It can be used for any High-Performance Computing (HPC) applications that leverage Massively Parallel Processing (MPP). The compute grid helps optimize overall cluster performance by collocating processing with data to optimize data processing and minimize network traffic. The system includes a comprehensive library of functions that includes machine and deep learning. Developers can develop and distribute their own code for any combination of transactions, analytics, stream processing or machine learning using Java,

.NET or C++. They can also leverage data affinity with collocated processing to achieve linear scalability as data sets grow.

The primary capabilities of the compute grid include:


  • Zero code (peer-class loading) deployment
  • Dynamic clustering
  • Fork-Join and MapReduce processing
  • Distributed closure execution
  • Load balancing and fault tolerance
  • Distributed messaging and events
  • Linear scalability
  • Standard Java ExecutorService support
  • Collocated processing support for multiple languages including Java, .NET and C++
  • DISTRIBUTED SQL


On top of this MPP architecture, Ignite provides ANSI-99 compliant, horizontally scalable distributed SQL. It supports all SQL, DDL and DML commands including SELECT, UPDATE, INSERT, MERGE and DELETE queries and CREATE and DROP table. Ignite supports distributed SQL joins. It allows for cross-cache joins across the entire cluster, which includes joins between data persisted in third-party databases and Ignite’s native persistence.

All processing, including SQL, is architected to collocate data and processing in a way that minimizes data movement across the network. Administrators can declare affinity keys such as foreign keys in DDL to partition data across the cluster. Distributed SQL joins are optimized with MPP techniques to take advantage of multi-table partitioning and replication. This helps ensure joins can happen with data locally on each node. Ignite can perform distributed SQL as real-time or batch across a single cluster that spans a host of third-party databases with SQL and NoSQL data.

ACID TRANSACTION SUPPORT

Ignite provides user-tunable ACID transaction support. You can define whether the system enforces strict or eventual consistency by using pessimistic or optimistic transactions. Strict consistency with pessimistic transactions supports high-value transaction use cases where consistency is more important than speed. Eventual consistency using optimistic transactions can be valuable for use cases where speed is paramount and the potential later rollback or correction of some transactions would be acceptable.

SERVICE GRID

Ignite provides a service grid to deploy and scale microservices across the cluster for digital business and other initiatives. It allows users to control how many instances of their service are deployed on each cluster – as a cluster singleton, node singleton, or as multiple instances across the cluster. The service grid guarantees continuous availability of all deployed services in case of node failures, including guaranteeing a single cluster or node singleton, or load balancing with multiple instances across the cluster.

Andy Pavlo - What Non-Volatile Memory Means for the Future of Database Management Systems



DISTRIBUTED MESSAGING AND EVENTS

Ignite provides high performance, cluster-wide messaging functionality to exchange data via publish-subscribe and direct point-to-point communication models. The primary capabilities of distributed messaging include support for topic-based publish-subscribe models and direct point-to-point communication, a pluggable communication transport layer, support for message ordering and cluster-aware message listener auto-deployment.

The distributed events functionality in Ignite allows applications to receive notifications about cache events occurring in a distributed grid environment. Developers can use this functionality to be notified about the execution of remote tasks or any cache data changes within the cluster. In Ignite, event notifications can be grouped together and sent in batches and/or timely intervals. Batching notifications help attain high cache performance and low latency.

The main capabilities of IMDG with Cassandra, Ignite improves read performance and adds ANSI-99 compliant SQL support to run any queries, including joins, aggregations, and groupings. Ignite can also be used integration with Apache Spark. Native support for Spark DataFrames allows Spark developers to access data from and save data to Ignite to share both data and state across Spark jobs. The Ignite RDD API lets developers read from and write to Ignite caches as mutable RDDs, unlike existing immutable Spark RDDs. Both the RDD and DataFrame support make Ignite caches accessible locally in RAM inside Spark processes executing Spark jobs.

Ignite also integrates its distributed SQL into SparkSQL plans. This allows Spark to take advantage of the advanced indexing and MPP-style distributed joins in Ignite. The combination can improve Spark SQL query performance by as much as 1000x.

The Ignite File System (IGFS) provides in-memory access via HDFS. Spark developers are able to leverage all of Ignite’s in-memory storage and processing capabilities including machine learning to train models in near real-time to improve outcomes for in-process HTAP applications.

In-Memory Computing Essentials for Architects and Engineers


IN-MEMORY HADOOP ACCELERATION

The Ignite accelerator for Hadoop enhances existing Hadoop environments by enabling fast data processing using the tools and technology your organization is already using today.

In-Memory Hadoop Acceleration in Ignite is based on the industry’s first dual-mode, high-performance in-memory file system that is 100% compatible with Hadoop HDFS and an in-memory optimized MapReduce implementation. In-memory HDFS and in-memory MapReduce provide easy to use extensions to disk-based HDFS and traditional MapReduce.

This plug-and-play feature requires minimal to no integration. It works with open source Hadoop or any  commercial version of Hadoop, including Cloudera®, HortonWorks®, MapR®, Intel®, AWS, as well as any other Hadoop 1.x or Hadoop 2.x distribution.

DISTRIBUTED IN-MEMORY FILE SYSTEM

One of the unique capabilities of Ignite is a file system interface to its in-memory data called the Ignite File System (IGFS). IGFS delivers similar functionality to Hadoop HDFS, including the ability to create a fully functional file system in memory. IGFS is at the core of the Ignite In-Memory Accelerator for Hadoop and can be plugged into any Hadoop or Spark environment.

The data from each file is split on separate data blocks and stored in cache. Developers can access the data in each file with a standard Java streaming API. For each part of the file, a developer can calculate an affinity and process the file’s content on corresponding nodes to avoid unnecessary networking.

WHAT GRIDGAIN® ADDS TO APACHE IGNITE

GridGain Features diagram
GridGain is the only enterprise-grade, commercially supported version of the Apache Ignite open source project. GridGain Systems contributed the code that became Ignite to the Apache Software Foundation and continues to be the project’s lead contributor. GridGain is 100% compatible with Ignite. GridGain adds enterprise-grade security, deployment, management and monitoring capabilities to Ignite. GridGain Systems also offers global support and professional services for business-critical systems. With Apache Ignite, patches are only released as part of each software release from the ASF, which happen every 3-6 months. Even though the community is committed to improving Ignite, there is no guarantee that a critical patch you might need makes it into the next release. GridGain Systems provides commercial SLAs with rapid response times and the ability to provide software patches much faster as needed. GridGain Systems also offers a professional services organization that has assisted with deployments across a wide range of customer use cases to help ensure your success and speed up your in-memory computing deployment.

Troubleshooting Apache® Ignite™



Management and Monitoring: GridGain Web Console, the GUI-based Management and Monitoring tool, provides a unified operations, management and monitoring system for GridGain deployments. GridGain Web Console provides management and monitoring views into all aspects of GridGain operations. This includes HPC, Data Grid, Streaming, and Hadoop acceleration via standard dashboards, advanced charting of performance metrics, and grid health (telemetry) views, among many other features.
Enterprise-Grade Security: The GridGain Enterprise Edition includes enterprise-grade Security that provides extensible and customizable authentication and security capabilities. It includes both a Grid Authentication SPI and a Grid Secure Session SPI to satisfy a variety of security requirements.
Network Segmentation Protection: Network Segmentation Protection detects any network disruption within the grid to manage transactional data grids during a ‘split brain’ scenario. The options for handling these network occurrences are fully configurable to help ensure the best approach to recover from different types of network-related issues.
Rolling Production Updates: The Rolling Production Updates feature enables you to co-deploy multiple versions of GridGain and allow them to co-exist as you roll out new versions. This prevents downtime when performing software upgrades.
Data Center Replication: GridGain reliably replicates data on a per-cache basis across two or more regions connected by wide area networks. This allows geographically remote data centers to maintain consistent views of data. With GridGain reliability and predictability, Data Center Replication ensures business continuity and can be used as part of a disaster recovery plan. Data Center Replication integrates with your application so that caches marked for replication are automatically synchronized across the WAN link.

In memory computing principles by Mac Moore of GridGain



Oracle GoldenGate Integration: The Oracle GoldenGate integration in the GridGain Enterprise and Ultimate Editions provides real-time data integration and replication into a GridGain cluster from different environments. When configured, the GridGain in-memory computing platform will automatically receive updates from the connected source database, converting the data from a database relational model to cache objects.
Centralized Backup and Recovery Management: The GridGain Ultimate Edition provides centralized backup and recovery using either the GridGain Web Console or Snapshot Command Line Tool. You can perform, schedule and manage backups, and then recover to any point in time on any cluster using a combination of full and incremental snapshots with continuous archiving. This includes the ability to backup remotely using network backups, and then (re) deploy to a different cluster of any size anywhere on premise or in the cloud. You can also deploy backups to support testing in development, quality assurance (QA) and staging environments.
Full, Incremental and Continuous Backups: Within the GridGain Web Console or Snapshot Command Line Tool you can centrally perform or schedule full and incremental snapshots across a distributed cluster. You can then use them as backup and restore points for later recovery. You can also use continuous archives of write-ahead log (WAL) files to backup down to each transaction. The combination of full and incremental snapshots with continuous archiving helps ensure data is never lost.

Deploying Distributed Databases and In-Memory Computing Platforms with Kubernetes


Network Backups: With the GridGain Ultimate Edition, snapshots do not need to be stored locally on the same cluster machines used to handle the operational load. They can also be managed and stored remotely on-premise or in the cloud. When combined with the remote storage of continuous archives, this capability helps ensure a cluster can be quickly recovered even if an entire data center disappears.
Point-in-Time Recovery: You can quickly restore a GridGain Ultimate Edition cluster to any point in time through the combination of full and incremental backups with continuous archiving. Point-in-time recovery can be used to restore a system up to any change without having to manually resubmit or replay existing transactions that occurred following a full or incremental snapshot. Continuous archiving helps spread network loads to minimize peak network traffic. It also allows  recovery  to be more granular and up-to-date, which helps reduce overall downtime needed to restore a cluster to a current, valid state.

On Cloud Nine: How to be happy migrating your in-memory computing platform to the cloud



Heterogeneous Recovery: The GridGain Ultimate Edition also allows you to restore an existing cluster to another location, on-premise or in the cloud, with a different size and topology. GridGain already allows you to dynamically add nodes to a cluster for scalability, and create a hybrid cluster across any collection of nodes or datacenters on-premise or in the cloud. Heterogeneous Recovery enables you to rapidly bring up a different size cluster the moment an existing cluster goes down or bring a new cluster up so that you can take an existing cluster down. This helps reduce downtime and increase availability.

GRIDGAIN SUPPORT

GridGain is the only company to provide commercial support for Apache Ignite. GridGain Basic Support for Apache Ignite and the GridGain Community Edition includes timely access to professional support via web or email. The team can help troubleshoot performance or reliability issues and suggest workarounds or patches, if necessary. A two-hour initial consultation allows our support team to understand your current environment for more effective support in the future. The consultation helps identify issues and improve the performance or reliability of your deployment.

Standard Support is for companies deploying the GridGain Enterprise Edition or the GridGain Community Edition in production. With 24x7 support hours and web, email, and phone access, Standard Support is perfect for ongoing production deployments. An annual license to the GridGain Enterprise or Community Edition is available with the subscription.

Premium Support is for companies deploying the GridGain Enterprise Edition or the GridGain Ultimate Edition for mission-critical applications. Premium Support is available 24x7 with the fastest initial response time, more named support contacts than Standard Support, and web, email, and phone access. Premium Support is available with a license to the GridGain Enterprise or Ultimate Edition.

Henning Andersen - Using Lock-free and Wait-free In-memory Algorithms to Turbo-charge High Volume Data Management



GRIDGAIN SOFTWARE EDITIONS

GridGain software editions include different capabilities and levels of support to fit specific needs. They are subscription-based products included with GridGain Support.

The GridGain Community Edition is a binary build of Apache Ignite created by GridGain Systems for companies that want to run Apache Ignite in a production environment. It includes optional LGPL dependencies, such as Hibernate L2 cache integration and Geospatial Indexing. It benefits from ongoing QA testing by GridGain Systems engineers and contains bug fixes which have not yet been released in the Apache Ignite code base. It is suitable for small-scale deployments which do not require support for multiple datacenters, enhanced management, and monitoring capabilities, or enterprise-grade security.

The GridGain Enterprise Edition is for companies that plan to run GridGain as an IMDG in production. It includes all of the features of Apache Ignite plus enterprise-grade features including datacenter replication, enterprise-grade security, rolling upgrades, expanded management and monitoring capabilities, and more. The Enterprise Edition is extensively tested by GridGain Systems and is recommended for use in large-scale or mission-critical production deployments or environments with heightened security requirements.

Roman Shtykh - Apache Ignite as a Data Processing Hub



The GridGain Ultimate Edition is for companies that plan to run GridGain as an in-memory database (IMDB) in production. It enables users to put the GridGain Persistent Store into production with confidence. The Ultimate Edition includes all the features of the GridGain Enterprise Edition plus centralized backup and recovery management. You can perform full, incremental  and continuous backups locally or across  a network, and point-in-time recovery as well as heterogeneous recovery where you can restore a cluster to any location on-premise or in the cloud with a different size and topology.

The GridGain Enterprise and Ultimate Editions include all of the features in Apache Ignite and the GridGain Community Edition plus additional integration, security, deployment, monitoring, network segmentation protection and optimization, data center management, and high availability capabilities. The GridGain Enterprise Edition includes all of these features for mission-critical in-memory data grid use cases. For those using GridGain as an in-memory database leveraging the GridGain Persistent Store, the Ultimate Edition includes all of the Enterprise Edition features plus centralized backup and recovery.

Nikita Shamgunov - Propelling IoT Innovation with Predictive Analytics


SUMMARY

Many companies use Apache Ignite and GridGain as shared data and processing infrastructure across projects to deliver in-memory speed and unlimited scalability for transactions, analytics, hybrid transactional and analytical processing (HTAP) and streaming analytics. GridGain is the only enterprise-grade, commercially supported version of the Apache Ignite open source project. GridGain includes enterprise-grade security, deployment, management and monitoring capabilities which are not in Ignite. GridGain Systems also offers global support and professional services. Both GridGain and Ignite provide speed and scalability by sliding between existing application and data layers as an in-memory data grid (IMDG) with no rip-and-replace of the existing database. They enable companies to deliver high volume, low latency transactions, and analytics. GridGain and Ignite also simplify streaming and analytics by acting as a shared data store and compute engine with real-time stream ingestion, processing, streaming analytics and continuous learning.

More Information

http://www.dataviz.my/2018/09/13/in-memory-computing-essentials-for-architects-and-developer/

https://docs.oracle.com/en/database/oracle/oracle-database/12.2/inmem/intro-to-in-memory-column-store.html#GUID-BFA53515-7643-41E5-A296-654AB4A9F9E7

https://docs.oracle.com/en/database/oracle/oracle-database/12.2/inmem/in-memory-column-store-architecture.html#GUID-DD7106DB-0BCE-4251-B808-8341507FDFC7

https://www.oracle.com/webfolder/technetwork/tutorials/jdedwards/White%20Papers/PerformanceOracleDatabaseInMemoryEOneWP.pdf

https://www.oracle.com/webfolder/technetwork/tutorials/architecture-diagrams/19/pdf/db-19c-architecture.pdf

https://www.gigaspaces.com/blog/in-memory-computing/

https://go.oracle.com/LP=85738?elqCampaignId=225885&src1=:ow:o:p:mt:&intcmp=WWMK190916P00003:ow:o:p:mt:

https://omegacloud.typepad.com/my-blog/2013/05/in-memory-computing-so-what-about-the-memory.html

https://www.oracle.com/technetwork/database/exadata/exadata-x8-8-ds-5444364.pdf

https://www.gridgain.com/resources/in-memory-computing-resources

https://www.gridgain.com/resources/papers/introducing-gridgain-in-memory-computing-platform

https://www.gridgain.com/resources/blog/in-memory-compute-grid-explained

https://www.gridgain.com/technology/in-memory-computing-platform

https://www.gridgain.com/technology/overview

https://cs.ulb.ac.be/public/_media/teaching/ignite_2017.pdf