Overview of Proficy Historian Kafka Connect

Proficy Historian Kafka Connect significantly strengthens connectivity by enabling seamless real-time and historical time series data exchange between Proficy Historian and Kafka brokers, using the Apache Avro™ data format. It operates on a Pub/Sub model, similar to other Kafka connectors.

The Proficy Historian Kafka Connect has the following two important components: Historian Kafka Consumer, and Historian Kafka Producer.

You can configure the Proficy Historian Kafka Connect as either a producer, to publish Proficy Historian data to Kafka topics, or as a consumer, to subscribe to Kafka topics, retrieve data published by different Kafka producers, and store them in the Proficy Historian database.

Proficy Historian Kafka Connect and Kafka Broker Compatibility

The Proficy Historian Kafka Connect was tested with the Apache Kafka and Confluent Kafka brokers to ensure good performance and compatibility. While these two brokers were primarily tested, the Proficy Historian Kafka Connect also supports other Kafka brokers, providing users with the flexibility to select and use other Kafka brokers.

Benefits of Using the Proficy Historian Kafka Connect

Efficient Real-Time Data Streaming/Processing: The Proficy Historian Kafka Connect as a producer can connect with several Kafka brokers to stream data in real-time and process data streamed by other data sources through brokers.
Efficient On-demand Data Publishing: The Proficy Historian Kafka Connect as a producer can connect with several Kafka brokers to publish data based on on-demand query. The on-demand query option is a customized option available within Historian Kafka Producer component. To use this option, you must understand the supported on-demand queries, configure your Kafka client to consume from the appropriate topics added in the Historian Kafka Producer, and ensure the messages data is mapped correctly for those topics.
Scalability: In a distributed environment, it enables horizontal scaling by connecting to multiple data sources that support Kafka, accommodating high data throughput. This can be achieved by connecting to a data broker, eliminating complex networking setups.
High Availability: The Proficy Historian Kafka Connect can be configured as a consumer to subscribe to Kafka brokers, which automatically handle high availability through standby brokers and partitions. In case of a network failure in one broker, another broker will be ready to publish data.
Controlled Data Exchange: You have control over the topics to which you subscribe and the messages/data you wish to retrieve. You can also configure criteria for good and bad data and retrieve only those that qualify.
Schema Mapping: Most Kafka brokers support messages based on the Avro™ schema. The Proficy Historian Kafka Connect enables schema mapping, allowing you to map data, including its type and attributes, to the target source's schema. This ensures consistency and compatibility of data exchanges among various data sources.
Custom Schema: The Proficy Historian Kafka Connect enables users to design and create their own schemas, defining the structure, data types, and relationships according to their requirements.
Smart Tag Parsing: The Proficy Historian Kafka Connect as a producer can smartly parse tags and convert them into meaningful schema with subsets derived from the tags, and then publish the data. This improves data management and processing by providing extra context, making data easier to interpret and organize. For more information, see Tag Parsing by Historian Kafka Producer.
Supports Arrays of data in the Schema for a Topic: The Proficy Historian Kafka Connect supports adding multiple data items as an array in the schema for a topic. This helps in handling multiple data items in one topic and contributes to improved performance.
Supports Avro™ Union: The Proficy Historian Kafka Connect supports Avro™ union. That is, it allows fields to accept multiple data types, enhancing schema flexibility. This is useful where the data structure might vary over time or across different datasets. For instance, a field can consist an integer or a string, accommodating different data inputs.
Supports Data Attributes and Additional Tag Properties in the Schema for a Topic: The Proficy Historian Kafka Connect supports the inclusion of data attributes and additional tag properties in a topic's schema. This improves data management and processing by providing extra context, making data easier to interpret and organize. It also enables efficient data retrieval and more filtered queries, allowing users to focus on specific data subsets.
Supports Subscription to Multiple Topics: The Historian Kafka Consumer allows subscription to multiple topics. When adding a Historian Kafka Consumer instance, you configure it to subscribe to multiple topics, eliminating the need to create multiple instances for subscribing to different topics.

Supports Wildcard Characters while subscribing to Topics: In addition to the option to subscribe to multiple topics using single Historian Kafka Consumer, you can include wildcard characters while subscribing to topics.

Wildcard characters simplify topic subscriptions by allowing you to use a specify patterns instead of specifying each topic individually. For instance, if you have topics named "topic1," "topic2," "topic3," and so on, you can use a pattern like "topic*" to subscribe to all these topics at once. This is helpful when topics are more, and/or dynamically updated/created, as it reduces subscription management and automatically includes new topics that match the .

The table below lists all the supported wildcard characters and patterns.

Wildcard Pattern	Description	Example Matches	Does Not Match
^Prefix.*	Matches topics with "prefix".	prefix1, prefix_abc	otherprefix, abc
.*suffix$	Matches topics ending with "suffix".	abc_suffix, 123suffix	suffix_123, abc
^exact_name$	Matches topics with the exact name.	exact_name	exact_name1, name
^.*	Matches all topics.	anytopic, another_topic	None
^topic[0-9]$	Matches topics named "topic" followed by a single digit (0-9).	topic1, topic5	topic10, topics
^topic[0-9] {2,3}$	Matches topics named "topic" followed by 2 or 3 digits.	topic12, topic123	topic1, topic1234

Complex queries: You can perform complex queries like filtering, and more.

Important Points to Know

Currently, Compression is not supported by the Historian Kafka Connect. Therefore, attempting to connect with compression enabled may result in an error.