v2.4.2

2.4.2 — 2019-12-04

Fixes

• Fixed don't set interrupt flag again after catching interrupt exception in Pulsar Client #5643
• Fixed data is not deleted after expiration due to connected readers #5621
• Fixed the go client docs missing in website #5595
• Fixed schema def build error with protobuf schema #5569
• Fixed docs about reset cursor #5551
• Fixed repeated initialization of connectorsManager #5545
• Fixed Functions unnecessarily restart during FunctionRuntimeManager init phase #5527
• Fixed list non-persistent topics shows the persistent topics #5502
• Return after triggering callback with empty result #5500
• Fixed dispatcher skipping delivery of a batch during concurrent replays #5499
• Fixed reader_listener option for Python API #5487
• Fixed wrongly report "3600 messages have timed-out" #5477
• Fixed broken custom auth-provider that uses authenticationData #5462
• Fixed negative ack tracker constructor sequence #5453
• Fixed StringSchema static initialization #5445
• Fixed message corruption on OOM for batch messages #5443
• Fixed couple functions related integration tests #5434
• Fixed bug that namespace policies does not take effect due to NPE #5408
• Fixed race condition : Failed to read-more entries on dispatcher #5391
• Fixed potential deadlock that can occur in addConsumer #5371
• Fixed proxy to be able to re-send request body #5361
• Fixed pulsar can't load the customized SerDe #5357
• Fixed instability in Pulsar Function window integration test #5337
• Fixed bk write failure, use signals to resume writing #5322
• Fixed bad_weak_ptr error when closing producer #5315
• Fixed typo(massage->message) causing prometheus metrics display error #5299
• Fixed invalidate cache on zk-cache timeout #5298
• Fixed memory leak caused by not being executed ClientConnection destructor #5286
• Fixed producer blocked after send an over size message while batch enabled #5282
• Fixed race condition while triggering message redelivery after an ack-timeout event #5276
• Fixed behavior when getting a key from functions state that doesn't exist #5272
• Fixed Cmake to build _pulsar for osx #5263
• Fixed client backoff setting does not take effect #5261
• Fixed memory leak caused by deadline_timer holding object reference #5246
• Fixed in Message Deduplication that may cause incorrect client/broker interaction #5243
• Fixed bug that fails to search namespace bundle due to NPE #5191
• Fixed bug that message delivery stops after resetting cursor for failover subscription #5185
• Fixed exception type check order bug #5174
• Fixed spark receiver to account for all the consumer config options #5152
• Fixed broker fails to start with function worker enabled and broker client using TLS #5151
• Fixed deadlock when resetting cursor #5136
• Fixed windowed functions were broken when we changed java function instance to use classloaders #5117
• Fixed storage size always 0 without subscription #5108

Enhancements

• Add subscribe position param for consumer of sink #5532
• Efficiency improvements for delay delivery tracker #5498
• Add is_read_compacted to create_reader() in python API #5483
• Make some common use method of ManagedLedger public #5472
• Avoid leak on publish failure on batch message #5442
• Record message failure and avoid exiting from process on publish failure #5441
• Add support for partitioned topic consumer seek by time #5435
• Add default loader for latest pyyaml #5432
• Trim messages which less than mark delete position for message redelivery #5378
• Make skip all messages async #5375
• Set default ensemble size to 2 in service conf, to match broker.conf #5339
• Only seek when reading unexpected entry #5356
• Don't require both region and endpoint to be specified #5355
• If cursor is not durable, close dispatcher when all consumers are removed from subscription #5340
• Disable stickyRead by default #5321
• Allow to specify delivery delay in C++ client #5317
• Add debug log + fix thread-factory name #5302
• Don't attempt to append on read-only cursor ledger #5297
• Close previous dispatcher when subscription type changes #5288
• Improve error handling logic for effectively once #5271
• Upgrade dependencies for security fixes #5232
• Allow for topic deletions with regex consumers #5230
• Ensure consumer background tasks are cancelled after subscribe failures #5204
• Added missing enum value KeyShared on the python wrapper #5196
• Make some member variables of Dispatcher volatile #5193
• Ensure getting list of topics for namespace is handled asynchronously #5188
• Close RateLimiter instance #5155
• Throw an error if the key was not specified for querying state #5145
• Allow configuring region aware placement related settings #5100
• DeleteBookieRack should remove the rack info from zookeeper #5084
• Use "info" as the default root logger level #5079
• Modify BatcherBuilder interface and it's subs to implement java.io.Serializable, otherwise java.io.NotSerializableException occurs when we use plusar-flink-connector #5068
• Don't return DEFAULT_RACK if ZkBookieRackAffinityMapping can't resolve network location #5067
• Reload zk cache asynchronously #5049
• Add different cache flags to ConcurrentOpenLongPairRangeSet for size() and toString() #5040
• Introduce number of threads in perf producer program #5036
• Completing connector configuration #4999
• Add checkstyle validation and fix style violations in the common module #4989
• Trim deleted entries after recover cursor #4987
• Expose getLastMessageId method in ConsumerImpl #4911
• Add a documentation page for metrics reference #4910
• Provide a convenient method for C++ client producer batch container #4885
• Add schema admin api get schema info with schema version #4877
• Return Message ID for send for cpp and cgo client #4811
• Add the schema admin api #4800
  ...

v2.4.1 release

2.4.1 — 2019-08-30

Fixes

• Fixed wrong serialize of batchBuilder in ProducerConfigurationData #4620
• Fixed SchemaInfo properties losing when registering schema using admin api #4617
• Fixed wrong topic domain returned by get partitioned topic #4613
• Fixed pulsar-dashboard wrong count number of bundles #4637
• Fixed partitionIndex error in consumer for a single partitioned topic #4591
• Fixed deadlock on get-status rest-api call in broker #4616
• Fixed deadlock in subscribeAsync demo #4649
• Fixed C++ client lookup error over HTTP in standalone #4625
• Fixed NPE at managed-ledger when fetch reader internal-stats #4615
• Fixed C++ client producer sendAsync() hang when no enough batched message #4657
• Fixed issue when submitting NAR via file url #4577
• Renamed C++ logger enum names to avoid conflicts with compiler macros #4664
• Fixed leaking of pulsar-io-influxdb in distribution #4678
• Fixed the default port for https and http in admin client #4623
• Fixed wrong schema delete when checking compatibility #4669
• Fixed docker/publish.sh doesn't work for pulsar-all and pulsar-standalone images #4705
• Fixed integration-test failure when execute pip3 install pulsar_client #4754
• Added log folder in pulsa-function-go back #4736
• Fixed Pulsar SQL NPE when predicate pushdown for publish_time #4744
• Fixed redelivered message logic of partition topic #4653
• Fixed C++ log level names in Log4cxxLogger #4735
• Fixed go function package not executable #4743
• Added anonymous role to proxy configuration #4733
• Added kubernetes namespace to function instance url #4701
• Fixed go function not parse conf content first #4746
• Made PulsarKafkaProducer thread safe #4745
• Fixed messages not get acked if there is not sink topic #4815
• Fixed bug when function package jar/py/go and runtime is not set #4814
• Fixed topic loading in rest-api not time out in zooKeeperOperationTimeoutSeconds #4805
• Fixed default retention policy miss upload to zk #4810
• Added checking function implements correct interface #4844
• Fixed retention size policy bug #4825
• Catch throwable in interceptors of consumer and producer #4860
• Fixed first position in managedLedger is ahead of the last #4853
• Fixed concurrent access of uninitializedCursors in ManagedLedgerImpl.asyncOpenCursor #4837
• Fixed schema not found handling in pulsar-sql #4890
• Fixed requests not respect no_proxy env variable in dashboard #4867
• Fixed broken replication msg to specific cluster #4930
• Fixed dashboard peek parse message error #4918
• Fixed StructSchema reader cache loading logic #4962
• Fixed fd leakage in FunctionActioner.downloadFile #4970
• Fixed python function str bytes convert in example custom_object_function.py #4946
• Fixed NPE while cleaning up namespace node #4965
• Fixed the getSchema logic in pulsar proxy #4975
• Fixed warning by add default loader for latest pyyaml #4974
• Fixed snappy compressor compile error in pulsar-client-cpp #4972
• Reinitialize certain components for externally managed runtimes when moving functions #5007
• Upgraded jackson-databind #5011
• Fixed the problem of missing dependencies less in docker #5034
• Fixed duplicated Prometheus TYPE in broker metrics #4183
• Fixed pulsar sink and source state by init state earlier #5046

Enhancements

• Re-factored Component implementation #4541
• Provided a clock for generating publish timestamp for producers #4562
• Reduced unnecessary track message calls #4595
• Supported dynamic configure with escape char #4611
• Added null check for function/source/sink configs #4627
• Added delete dynamic config api #4614
• Made broker replication mtls configuration dynamic #4609
• Added authorization to function worker REST endpoints #4628
• Improved and add authorization to function download and upload #4644
• Allowed consumer retrieve the sequence id that the producer set #4645
• Added perPartition parameter to partitioned-stats API #4639
• Supported Pulsar schema for pulsar kafka client wrapper #4534
• Supported delete and update event for JDBC Sink #4358
• Cleaned up tests in the presto module #4683
• Added allowAutoTopicCreation to broker.conf and related configuration #4694
• Changed to use classloaders to load Java functions #4685
• Removed fixed server type check in kerberos #4758
• Changed type of publish_time to timestamp #4757
• Added read-timeout to admin-request #4762
• Added checking of deleted schema when adding schema #4731
• Added strand to C++ client for exclusive control #4750
• Added support to create partitioned topic with 1 partition #4764
• Added getters and setters to PulsarService & BrokerService #4709
• Added configure ack-timeout tick time #4760
• Added options to rewrite namespace delimiter for pulsar sql #4749
• Made Pulsar SQL supports pulsar's primitive schema #4728
• Added basic authentication capabilities to Pulsar SQL #4779
• Improved SchemaInfoProvider to fetch schema info asynchronously #4836
• Supported KeyValue schema use AUTO_CONSUME as key/value schema #4839
• Provided action type for insert in pulsar-io-jdbc #4862
• Made partition as internal column in pulsar-sql #4888
• Added option to disable authentication for proxy /metrics #4921

v2.4.0

Apache

2.4.0 — 2019-06-30

PIPs

• PIP-26: Added support for delayed message delivery
  #4062, note that messages are only delayed on shared subscriptions.
• PIP-28: Pulsar Proxy Gateway Improvement #3915
• PIP-29: One package for both pulsar client and pulsar admin #3662
• PIP-30: Change authentication provider API to support mutual authentication for Kerberos Authentication #3677
  #3821 #3997
• PIP-32: Go Function API, Instance and LocalRun #3854 #4008 #4174
• PIP-33: Replicated subscriptions to keep subscription state in-sync across multiple geographical regions #4299 #4340 #4354 #4396
• PIP-34: Key_Shared subscription, Architecture for Key_Shared #4079 #4120 #4335 #4372 #4407 #4406 4462
• PIP-36: Support set message size in broker.conf #4247

Fixes

• Fix possible message loss using peer-cluster feature #3426
• Fix pulsar standalone does not read zk port from conf/standalone.conf #3790
• Fix some issues of ZkIsolatedBookieEnsemblePlacementPolicy #3917 #3918
• Fix NPE when unload non-existent topic #3946
• Fix race condition while deleting global topic #4173
• Fix deadlock on skip messages #4411
• Fix NPE when closing batch during a reconnection #4427
• Fix race condition of read-timeout task in managed ledger #4437
• Disable sticky read by default #4526
• Fix race condition between timeout-task and add-call complete #4455

Enhancements

• Optimize message replay for large backlog consumer #3732
• Added support for websocket produce/consume command #3835
• Added support for TTL config in broker.conf #3898
• Reduce memory used in ClientCnx for pending lookups #4104
• Reduce number of hashmap sections for ledger handles cache #4102
• Added backlog and offloaded size in Prometheus stats #4150
• Added support for configure the managed ledger cache eviction frequency #4066
• Added support to avoid payload copy when inserting into managed ledger cache #4197
• Added support to cache unack-messageId into OpenRangeSet #3819
• Added support configure static PulsarByteBufAllocator to handle OOM errors #4196
• Auto refresh new tls certs for jetty webserver #3645
• Create non-persistent topic by pulsar-admin/rest api #3625
• Consumer priority-level in Failover subscription #2954

Security

• Added support for other algorithms in token auth #4528

Namespace Policies

• Added support for tenant based bookie isolation #3933
• Added support for secondary bookie isolation group at namespace #4458
• Added support for secondary bookie-isolation-group #4261
• Added support for replicator rate limit between clusters #4273
• Disable backlog quota check by default #4320

Tiered Storage

• Added support for Pulsar SQL to read data from tiered storage #4045

Pulsar Schema

• Added schema versioning to support multi version messages produce and consume #3876 #3670 #4211 #4325 #4548
• Added TRANSITIVE schema check strategies to support compatibility check over all existing schemas #4214
• Added schema data validator #4360
• Added support for delete schema when deleting a topic #3941
• Added generic record builder #3690

Pulsar IO

• Added IO connector for flume source and sink #3597
• Added IO connector for redis sink #3700
• Added IO connector for solr sink #3885
• Hide kafka-connecter details for easy use debezium connector #3825
• Added IO connector for debezium PostgreSQL source #3924
• Enhancements for RabbitMQ source configuration #3937
• Added IO connector for RabbitMQ sink #3967
• Added IO connector for InfluxDB sink #4017

Pulsar functions

• Added support for authentication #3735 #3874 #4198
• Fix NPE when stats manager not initialized #3891
• Added async state manipulation methods #3798
• Fix fail to update functions in effectively-once mode #3993
• Added labels to function statefulsets and services #4038
• Added support for set key for message when using function publish #4005
• Use negative acknowledge to instead ackTimeout #4103
• Fix backward compatibility with 2.2 auth not working #4241

Java client

• Added negative acks #3703
• Added support for backoff strategy configuration #3848
• Added support for configure TypedMessageBuilder through a Map conf object #4015
• Added interceptor for negative ack send #3962
• Added support for seek operate on reader #4031
• Store key part of KeyValue schema into pulsar message keys #4117
• Added interceptor for ack timeout #4300
• Added support for snappy compression #4259
• Added support for key based batcher #4435

Python client

• Added negative acks #3816
• Added support for snappy compression #4319

Go client

• Added negative acks #3817
• Added support for go schema #3904
• Added support for snappy compression #4319
• Added support for Key_Shared subscription [#4465](https://github.com/apache/pulsar/pul...

v2.3.2

2.3.2 — 2019-05-30

Fixes

• Validate admin operation on topic with authoritative parameter #4270
• fix bug with source local run #4278
• fix cannot use size (type _Ctype_int) as type _Ctype_ulong #4212
• Fix segfault in c++ producer #4219
• AlwaysCompatible doesn't use AlwaysSchemaValidator in 2.3.1 component/schemaregistry #4181
• Avoid potentially blocking method during topic ownership check #4190
• [pulsar-broker]Fix: client-producer can't reconnect due to failed producer-future on cnx cache #4138
• Removing # TYPE comment from topic metrics in Prometheus #4136
• For functions metrics in prometheus also remove TYPE #4081
• Fix: set receive queue size for sinks #4091
• Fix: Exception when switch cluster from auth enabled to auth disabled #4069
• Fix update cli source sink #4061
• Fix connectors nested configs #4067
• For functions metrics, avoid having HELP #4029
• Fix Python functions state which is completely broken #4027
• [issue #3975] Bugfix NPE on non durable consumer #3988
• Fix: Function auth should ignore exception because it might be anonymous user #4185
• [pulsar-function] fix worker-stats broken admin-api #4068
• fix errors in sql doc #4030
• Fix the swagger files generated by removing troublesome class #4024
• [pulsar-function] fix broken backward compatibility with v1-namespace while registering function #4224
• Revert dup consumer and related code #4142
• [issue 4274][pulsar-io]Add double quotation marks for metrics with remote_cluster #4295

Enhancements

• By default, auto configure the size of Bookie read/write cache #4297
• Upgrade to BookKeeper 4.9.2 #4288
• [pulsar-function] support bookie authentication from function-worker #4088
• Optimizing performance for Pulsar function archive download #4082
• allow users to update output topics for functions and sources #4092
• improve data-generator source performance #4058
• [client] Set actual topic name to partitioned consumer #4064
• ack records in datagenerator print sink #4052
• [security] Upgrade athenz libraries #4056
• [python client] Handle subrecords in JsonSchema encoding #4023
• [Issue 3987][pulsar-broker]Handle config is null when create tenant #4019
• Add bookkeeper client version constraint #4013
• Improve error handling for triggering function when there is a schema mismatch #3995
• [pulsar-broker] add producer/consumer id in error-logging #3961

2.3.1

Fixes

• Fixed C++ batch acks tracker to evict message from sendList array. This was causing a slowdown in
  C++ consumers #3618
• Allow publishing messages >5MB with batching (when they compress to <5MB) #3673 and #3718
• Use at least 8 threads in Jetty thread pool. This fixed deadlocks in Jetty requests handling #3776
• Fixed Reader.HasNext() in Go client #3764
• Fixed increasing consumer permits after ack dedup operation. #3787
• Set the dedup cursor as "inactive" after recovery #3612
• Fix read batching message by pulsar reader #3830
• Fix submit function with code specified via URL #3934
• Fixed reader reading from a partition #3960
• Fixed issue with Authorization header missing after client gets redirected #3869

Enhancements

• Added producer.flush() on Python #3685
• Introduced schema builder to define schema #3682
• Support passing schema definition for JSON and AVRO schemas #3766
• Exposing InitialPosition setting in Python consumer #3714

For a complete list of issues fixed, see

https://github.com/apache/pulsar/milestone/21?closed=1

2.3.0

General

• Support for schema definitions in the Pulsar Python client library
• PIP-25: Token based authentication #2888, #3067 and #3089
• Updated to Apache BookKeeper 4.9.0
• ZStandard compression codec #3159. Note that when a producer choose Zstd compression, a consumer will need to be at least at version 2.3.0
  to be able to correctly receive the messages.
• Support for Java 11 #3006
• Added Schema.AUTO_PRODUCE type to allow to publish serialized data and validate it against the
  topic schema #2685
• Added Schema.KeyValue to allow for schema to be validated on message keys as well as payloads. #2885
• Support TLS authentication and authorization in standalone mode #3360
• When creating namespace, use local cluster by default #3571
• Tag BookKeeper ledgers created by Pulsar with topic/subscription names for info/debug purposes
  #3525
• Enabled sticky reads in BooKeeper reads to increase IO efficiency with read-ahead #3569
• Several optimization in Pulsar SQL Presto connector (#3128, #3135, #3139, #3144, #3143)
• Configure Pulsar broker data cache automatically from JVM settings #3573
• Reuse the SSL context objects #3550
• Automatic schema update can be disabled through admin interface #2691
• Support Dead-Letter-Queue from WebSocket proxy #2968
• Pull-mode for WebSocket proxy #3058
• Export Jetty stats to Prometheus #2804
• Added stats for Pulsar proxy #2740
• Allow subscribers to access subscription admin-api #2981
• Make brokers read on closest Bookie in a multi-region deployment #3171

Fixes

• Fixed deadlock in reusing ZookKeeper event thread #3591
• In functions log topic appender, don't set producer name #3544
• When cursor recovery encounters empty cursor ledger, fallback to latest snapshot #3487
• Fixed C++ regex-consumer when using HTTP service URL #3407
• Fix race condition: broker not scheduling read for active consumer #3411

Pulsar IO

• Added Debezium connector for Change-Data-Capture into Pulsar #2791
• Added MongoDB connector #3561
• Added Elastic Search connector #2546
• Added HBase sink #3368
• Added Local files connector #2869
• Report source/sink stats in Prometheus #3261
• Allow filtering in Twitter Firehose connector #3298
• Sources/Sinks can be launched using fat jars as well #3166

Pulsar Functions

• Added Kubernetes runtime #1950
• Secrets interface #2826
• Cleanup subscriptions when deleting functions #3299
• Add Windowfunction interface to functions api #3324
• Support for accessing state in Python #2714
• Support submitting Python functions as wheel file
• Support submitting Python functions as Zip file with dependencies included #3321
• Add minimum amount of resources to run setting for functions #3536
• Fixed the behavior of Function start/stop #3477

Java client

• Moved Pulsar v1 client API into separate artifact #3228.
  
  Applications that are using the Pulsar v1 API, deprecated since 2.0 release, need to update the Maven dependency to use the pulsar-client-1x artifact instead of pulsar-client. Eg.

  <dependency>
      <groupId>org.apache.pulsar</groupId>
      <artifactId>pulsar-client-1x</artifactId>
      <version>2.3.0</version>
  </dependency>

• Fixed shading issues with Javadoc bundled in client jars by separating the API in a different Maven module #3309
• Improve Javadocs #3592
• Support specifying multiple hosts in pulsar service url and web url #3249
• Automatically discover when partitions on a topic are increased #3513
• Added Client.getPartitionsForTopic() #2972 (Javadoc)
• Added Consumer.pauseMessageListener() and Consumer.resumeMessageListener() #2961
• Removed shading relocations for Circe-checksum and lz4 libraries, to ensure native libraries
  are correctly loaded when using shaded client lib. #2191

Python client

• Fixed Message.properties() #3595

Go client

• Added Producer.flush() to flush all outstanding messages #3469
• Support Consumer.Seek() #3478
• Added Message.Topic() #3346
• Allow to specify SubscriptionInitPos option in ConsumerOptions #3588
• Added TLS hostname verification #3580
• Allow to link statically against libpulsar.a#3488
• Expose Producer.LastSequenceID() and Message.SequenceID() #3416

C++ client

• Enable batching by default when using sendAsync() #2949
• Allow to specify schema info in Avro format #3354
• Added Producer.flush() to flush all outstanding messages #3020
• Added TLS hostname verification #2475
• Allow to specify SubscriptionInitialPosition #3567
• Added Message.getTopicName() #3326
• Added Cosnsumer.receiveAsync() #3389
• Build libpulsar.a with all required dependencies #3488
• Removed Boost from Pulsar API headers #3374

Adaptors

• Kafka client wrapper, added support for explicit partitioning and custom partitioner #3462
• Support config auto.offset.reset to Pulsar KafkaConsumer #3273
• In Apache Flink connector, added support for Batch Sink API (2979#, #3039 and #3046)
• Added Java batch examples for Flink adaptor
• Added Java streaming examples for Flink adaptor
• Added Scala examples for Flink adaptor

For a complete list of issues fixed, see

https://github.com/apache/pulsar/milestone/18?closed=1

Downloads at https://pulsar.apache.org/download

2.2.1

This release includes fixes for 2.2.0 release. In particular:

• Fixed issue when proxy HTTP admin API requests Pulsar proxy #3022
• Fixed Consumer.unsubscribe() in Python client library #3093
• Fixed ZLib decompression in C++ client #2854
• Fixed Pulsar functions context publish in Python #2844

For a complete list of issues fixed, see https://github.com/apache/pulsar/milestone/19?closed=1

2.2.0

This is the first release of Pulsar as an Apache Top Level Project

This is a feature release, including several new features, improvements and fixes for issues reported for 2.1.1-incubating.

• Pulsar SQL
• Pulsar Java Client Interceptors
• Integration of functions and io with schema registry
• Dead Letter Topic
• Flink Source connector
• JDBC Sink Connector
• HDFS Sink Connector
• Google Cloud Storage Offloader

2.1.1-incubating

This release fixes issues reported for 2.1.0-incubating.

• #2473 - Downgrading ZK to stable version 3.4.13
• #2219 - Cpp client: add PatternMultiTopicsConsumerImpl to support regex subscribe
• #2387 - Fixed race condition during expansion of concurrent open hash maps
• #2348 - Fix NPE when splitting and unloading bundle
• #2223 - fix bug in FunctionRuntimeManager involving not cleaning up old invalid assignments
• #2367 - [compaction] make topic compaction works with partitioned topic
• #2203 - Make sure schema is initialized before the topic is loaded

The complete list of changes can be found at:

https://github.com/apache/incubator-pulsar/milestone/17?closed=1

2.1.0-incubating

This is the seventh release of Apache Pulsar since entering the ASF incubator.

There are several new features and major improvements:

• Pulsar IO: A connector framework for moving data in and out of Apache Pulsar leveraging Pulsar Functions runtime.
• Tiered Storage: An extension in Pulsar segment store to offload older segments into long term storage (e.g. HDFS, S3). S3 support is built in 2.1 release.
• Stateful function (preview) in Pulsar Functions
• Pulsar Go Client
• Avro and Protobuf Schema support

The complete list of changes can be found at: https://github.com/apache/incubator-pulsar/milestone/13?closed=1