GSaha567/seq_level_training_data

1text,length,is_long_context,metric_val,label_metric2"[[netty-http-component]]3= Netty HTTP Component4:docTitle: Netty HTTP5:artifactId: camel-netty-http6:description: Netty HTTP server and client using the Netty 4.x.7:since: 2.148:supportLevel: Stable9:component-header: Both producer and consumer are supported10include::{cq-version}@camel-quarkus:ROOT:partial$reference/components/netty-http.adoc[opts=optional]11 12*Since Camel {since}*13 14*{component-header}*15 16The Netty HTTP component is an extension to xref:netty-component.adoc[Netty]17component to facilitiate HTTP transport with xref:netty-component.adoc[Netty].18 19 20[NOTE]21====22*Stream*23 24Netty is stream based, which means the input it receives is submitted to25Camel as a stream. That means you will only be able to read the content26of the stream *once*. If you find a situation where the message body appears to be empty or27you need to access the data multiple times (eg: doing multicasting, or28redelivery error handling) you should use Stream caching or convert the29message body to a `String` which is safe to be re-read multiple times. 30Notice Netty HTTP reads the entire stream into memory using31`io.netty.handler.codec.http.HttpObjectAggregator` to build the entire32full http message. But the resulting message is still a stream based33message which is readable once.34====35 36Maven users will need to add the following dependency to their `pom.xml`37for this component:38 39[source,xml]40------------------------------------------------------------41<dependency>42    <groupId>org.apache.camel</groupId>43    <artifactId>camel-netty-http</artifactId>44    <version>x.x.x</version>45    <!-- use the same version as your Camel core version -->46</dependency>47------------------------------------------------------------48 49== URI format50 51The URI scheme for a netty component is as follows52 53-------------------------------------------54netty-http:http://0.0.0.0:8080[?options]55-------------------------------------------56 57[NOTE]58====59*Query parameters vs endpoint options*60 61You may be wondering how Camel recognizes URI query parameters and62endpoint options. For example you might create endpoint URI as follows:63`netty-http:http//example.com?myParam=myValue&compression=true` . In64this example `myParam` is the HTTP parameter, while `compression` is the65Camel endpoint option. The strategy used by Camel in such situations is66to resolve available endpoint options and remove them from the URI. It67means that for the discussed example, the HTTP request sent by Netty68HTTP producer to the endpoint will look as follows:69`http//example.com?myParam=myValue`, because `compression` endpoint70option will be resolved and removed from the target URL.71Keep also in mind that you cannot specify endpoint options using dynamic72headers (like `CamelHttpQuery`). Endpoint options can be specified only73at the endpoint URI definition level (like `to` or `from` DSL elements).74====75 76[IMPORTANT]77====78*A lot more options*79 80This component inherits all the options from81xref:netty-component.adoc[Netty], so make sure to look at82the xref:netty-component.adoc[Netty] documentation as well.83Notice that some options from xref:netty-component.adoc[Netty] is not84applicable when using this Netty HTTP component, such as options85related to UDP transport.86====87 88// component-configure options: START89== Configuring Options90 91Camel components are configured on two separate levels:92 93- component level94- endpoint level95 96=== Configuring Component Options97 98The component level is the highest level which holds general and common configurations that are inherited by the endpoints.99For example a component may have security settings, credentials for authentication, urls for network connection and so forth.100 101Some components only have a few options, and others may have many. Because components typically have pre configured defaults102that are commonly used, then you may often only need to configure a few options on a component; or none at all.103 104Configuring components can be done with the xref:latest@manual::component-dsl.adoc[Component DSL],105in a configuration file (application.properties|yaml), or directly with Java code.106 107=== Configuring Endpoint Options108 109Where you find yourself configuring the most is on endpoints, as endpoints often have many options, which allows you to110configure what you need the endpoint to do. The options are also categorized into whether the endpoint is used as consumer (from)111or as a producer (to), or used for both.112 113Configuring endpoints is most often done directly in the endpoint URI as path and query parameters. You can also use114the xref:latest@manual::Endpoint-dsl.adoc[Endpoint DSL] as a _type safe_ way of configuring endpoints.115 116A good practice when configuring options is to use xref:latest@manual::using-propertyplaceholder.adoc[Property Placeholders],117which allows to not hardcode urls, port numbers, sensitive information, and other settings.118In other words placeholders allows to externalize the configuration from your code, and gives more flexibility and reuse.119 120The following two sections lists all the options, firstly for the component followed by the endpoint.121// component-configure options: END122 123// component options: START124== Component Options125 126 127The Netty HTTP component supports 76 options, which are listed below.128 129 130 131[width=""100%"",cols=""2,5,^1,2"",options=""header""]132|===133| Name | Description | Default | Type134| *configuration* (common) | To use the NettyConfiguration as configuration when creating endpoints. |  | NettyConfiguration135| *disconnect* (common) | Whether or not to disconnect(close) from Netty Channel right after use. Can be used for both consumer and producer. | false | boolean136| *keepAlive* (common) | Setting to ensure socket is not closed due to inactivity | true | boolean137| *reuseAddress* (common) | Setting to facilitate socket multiplexing | true | boolean138| *reuseChannel* (common) | This option allows producers and consumers (in client mode) to reuse the same Netty Channel for the lifecycle of processing the Exchange. This is useful if you need to call a server multiple times in a Camel route and want to use the same network connection. When using this, the channel is not returned to the connection pool until the Exchange is done; or disconnected if the disconnect option is set to true. The reused Channel is stored on the Exchange as an exchange property with the key NettyConstants#NETTY_CHANNEL which allows you to obtain the channel during routing and use it as well. | false | boolean139| *sync* (common) | Setting to set endpoint as one-way or request-response | true | boolean140| *tcpNoDelay* (common) | Setting to improve TCP protocol performance | true | boolean141| *bridgeErrorHandler* (consumer) | Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions occurred while the consumer is trying to pickup incoming messages, or the likes, will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions, that will be logged at WARN or ERROR level and ignored. | false | boolean142| *broadcast* (consumer) | Setting to choose Multicast over UDP | false | boolean143| *clientMode* (consumer) | If the clientMode is true, netty consumer will connect the address as a TCP client. | false | boolean144| *reconnect* (consumer) | Used only in clientMode in consumer, the consumer will attempt to reconnect on disconnection if this is enabled | true | boolean145| *reconnectInterval* (consumer) | Used if reconnect and clientMode is enabled. The interval in milli seconds to attempt reconnection | 10000 | int146| *backlog* (consumer) | Allows to configure a backlog for netty consumer (server). Note the backlog is just a best effort depending on the OS. Setting this option to a value such as 200, 500 or 1000, tells the TCP stack how long the accept queue can be If this option is not configured, then the backlog depends on OS setting. |  | int147| *bossCount* (consumer) | When netty works on nio mode, it uses default bossCount parameter from Netty, which is 1. User can use this option to override the default bossCount from Netty | 1 | int148| *bossGroup* (consumer) | Set the BossGroup which could be used for handling the new connection of the server side across the NettyEndpoint |  | EventLoopGroup149| *disconnectOnNoReply* (consumer) | If sync is enabled then this option dictates NettyConsumer if it should disconnect where there is no reply to send back. | true | boolean150| *executorService* (consumer) | To use the given EventExecutorGroup. |  | EventExecutorGroup151| *maximumPoolSize* (consumer) | Sets a maximum thread pool size for the netty consumer ordered thread pool. The default size is 2 x cpu_core plus 1. Setting this value to eg 10 will then use 10 threads unless 2 x cpu_core plus 1 is a higher value, which then will override and be used. For example if there are 8 cores, then the consumer thread pool will be 17. This thread pool is used to route messages received from Netty by Camel. We use a separate thread pool to ensure ordering of messages and also in case some messages will block, then nettys worker threads (event loop) wont be affected. |  | int152| *nettyServerBootstrapFactory* (consumer) | To use a custom NettyServerBootstrapFactory |  | NettyServerBootstrapFactory153| *networkInterface* (consumer) | When using UDP then this option can be used to specify a network interface by its name, such as eth0 to join a multicast group. |  | String154| *noReplyLogLevel* (consumer) | If sync is enabled this option dictates NettyConsumer which logging level to use when logging a there is no reply to send back. There are 6 enums and the value can be one of: TRACE, DEBUG, INFO, WARN, ERROR, OFF | WARN | LoggingLevel155| *serverClosedChannelException{zwsp}CaughtLogLevel* (consumer) | If the server (NettyConsumer) catches an java.nio.channels.ClosedChannelException then its logged using this logging level. This is used to avoid logging the closed channel exceptions, as clients can disconnect abruptly and then cause a flood of closed exceptions in the Netty server. There are 6 enums and the value can be one of: TRACE, DEBUG, INFO, WARN, ERROR, OFF | DEBUG | LoggingLevel156| *serverExceptionCaughtLogLevel* (consumer) | If the server (NettyConsumer) catches an exception then its logged using this logging level. There are 6 enums and the value can be one of: TRACE, DEBUG, INFO, WARN, ERROR, OFF | WARN | LoggingLevel157| *serverInitializerFactory* (consumer) | To use a custom ServerInitializerFactory |  | ServerInitializerFactory158| *usingExecutorService* (consumer) | Whether to use ordered thread pool, to ensure events are processed orderly on the same channel. | true | boolean159| *connectTimeout* (producer) | Time to wait for a socket connection to be available. Value is in milliseconds. | 10000 | int160| *lazyStartProducer* (producer) | Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel's routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing. | false | boolean161| *requestTimeout* (producer) | Allows to use a timeout for the Netty producer when calling a remote server. By default no timeout is in use. The value is in milli seconds, so eg 30000 is 30 seconds. The requestTimeout is using Netty's ReadTimeoutHandler to trigger the timeout. |  | long162| *clientInitializerFactory* (producer) | To use a custom ClientInitializerFactory |  | ClientInitializerFactory163| *correlationManager* (producer) | To use a custom correlation manager to manage how request and reply messages are mapped when using request/reply with the netty producer. This should only be used if you have a way to map requests together with replies such as if there is correlation ids in both the request and reply messages. This can be used if you want to multiplex concurrent messages on the same channel (aka connection) in netty. When doing this you must have a way to correlate the request and reply messages so you can store the right reply on the inflight Camel Exchange before its continued routed. We recommend extending the TimeoutCorrelationManagerSupport when you build custom correlation managers. This provides support for timeout and other complexities you otherwise would need to implement as well. See also the producerPoolEnabled option for more details. |  | NettyCamelStateCorrelationManager164| *lazyChannelCreation* (producer) | Channels can be lazily created to avoid exceptions, if the remote server is not up and running when the Camel producer is started. | true | boolean165| *producerPoolEnabled* (producer) | Whether producer pool is enabled or not. Important: If you turn this off then a single shared connection is used for the producer, also if you are doing request/reply. That means there is a potential issue with interleaved responses if replies comes back out-of-order. Therefore you need to have a correlation id in both the request and reply messages so you can properly correlate the replies to the Camel callback that is responsible for continue processing the message in Camel. To do this you need to implement NettyCamelStateCorrelationManager as correlation manager and configure it via the correlationManager option. See also the correlationManager option for more details. | true | boolean166| *producerPoolMaxActive* (producer) | Sets the cap on the number of objects that can be allocated by the pool (checked out to clients, or idle awaiting checkout) at a given time. Use a negative value for no limit. | -1 | int167| *producerPoolMaxIdle* (producer) | Sets the cap on the number of idle instances in the pool. | 100 | int168| *producerPoolMinEvictableIdle* (producer) | Sets the minimum amount of time (value in millis) an object may sit idle in the pool before it is eligible for eviction by the idle object evictor. | 300000 | long169| *producerPoolMinIdle* (producer) | Sets the minimum number of instances allowed in the producer pool before the evictor thread (if active) spawns new objects. |  | int170| *udpConnectionlessSending* (producer) | This option supports connection less udp sending which is a real fire and forget. A connected udp send receive the PortUnreachableException if no one is listen on the receiving port. | false | boolean171| *useByteBuf* (producer) | If the useByteBuf is true, netty producer will turn the message body into ByteBuf before sending it out. | false | boolean172| *hostnameVerification* ( security) | To enable/disable hostname verification on SSLEngine | false | boolean173| *allowSerializedHeaders* (advanced) | Only used for TCP when transferExchange is true. When set to true, serializable objects in headers and properties will be added to the exchange. Otherwise Camel will exclude any non-serializable objects and log it at WARN level. | false | boolean174| *autowiredEnabled* (advanced) | Whether autowiring is enabled. This is used for automatic autowiring options (the option must be marked as autowired) by looking up in the registry to find if there is a single instance of matching type, which then gets configured on the component. This can be used for automatic configuring JDBC data sources, JMS connection factories, AWS Clients, etc. | true | boolean175| *channelGroup* (advanced) | To use a explicit ChannelGroup. |  | ChannelGroup176| *headerFilterStrategy* (advanced) | To use a custom org.apache.camel.spi.HeaderFilterStrategy to filter headers. |  | HeaderFilterStrategy177| *nativeTransport* (advanced) | Whether to use native transport instead of NIO. Native transport takes advantage of the host operating system and is only supported on some platforms. You need to add the netty JAR for the host operating system you are using. See more details at: \\http://netty.io/wiki/native-transports.html | false | boolean178| *nettyHttpBinding* (advanced) | To use a custom org.apache.camel.component.netty.http.NettyHttpBinding for binding to/from Netty and Camel Message API. |  | NettyHttpBinding179| *options* (advanced) | Allows to configure additional netty options using option. as prefix. For example option.child.keepAlive=false to set the netty option child.keepAlive=false. See the Netty documentation for possible options that can be used. |  | Map180| *receiveBufferSize* (advanced) | The TCP/UDP buffer sizes to be used during inbound communication. Size is bytes. | 65536 | int181| *receiveBufferSizePredictor* (advanced) | Configures the buffer size predictor. See details at Jetty documentation and this mail thread. |  | int182| *sendBufferSize* (advanced) | The TCP/UDP buffer sizes to be used during outbound communication. Size is bytes. | 65536 | int183| *transferExchange* (advanced) | Only used for TCP. You can transfer the exchange over the wire instead of just the body. The following fields are transferred: In body, Out body, fault body, In headers, Out headers, fault headers, exchange properties, exchange exception. This requires that the objects are serializable. Camel will exclude any non-serializable objects and log it at WARN level. | false | boolean184| *udpByteArrayCodec* (advanced) | For UDP only. If enabled the using byte array codec instead of Java serialization protocol. | false | boolean185| *workerCount* (advanced) | When netty works on nio mode, it uses default workerCount parameter from Netty (which is cpu_core_threads x 2). User can use this option to override the default workerCount from Netty. |  | int186| *workerGroup* (advanced) | To use a explicit EventLoopGroup as the boss thread pool. For example to share a thread pool with multiple consumers or producers. By default each consumer or producer has their own worker pool with 2 x cpu count core threads. |  | EventLoopGroup187| *allowDefaultCodec* (codec) | The netty component installs a default codec if both, encoder/decoder is null and textline is false. Setting allowDefaultCodec to false prevents the netty component from installing a default codec as the first element in the filter chain. | true | boolean188| *autoAppendDelimiter* (codec) | Whether or not to auto append missing end delimiter when sending using the textline codec. | true | boolean189| *decoderMaxLineLength* (codec) | The max line length to use for the textline codec. | 1024 | int190| *decoders* (codec) | A list of decoders to be used. You can use a String which have values separated by comma, and have the values be looked up in the Registry. Just remember to prefix the value with # so Camel knows it should lookup. |  | List191| *delimiter* (codec) | The delimiter to use for the textline codec. Possible values are LINE and NULL. There are 2 enums and the value can be one of: LINE, NULL | LINE | TextLineDelimiter192| *encoders* (codec) | A list of encoders to be used. You can use a String which have values separated by comma, and have the values be looked up in the Registry. Just remember to prefix the value with # so Camel knows it should lookup. |  | List193| *encoding* (codec) | The encoding (a charset name) to use for the textline codec. If not provided, Camel will use the JVM default Charset. |  | String194| *textline* (codec) | Only used for TCP. If no codec is specified, you can use this flag to indicate a text line based codec; if not specified or the value is false, then Object Serialization is assumed over TCP - however only Strings are allowed to be serialized by default. | false | boolean195| *enabledProtocols* (security) | Which protocols to enable when using SSL | TLSv1,TLSv1.1,TLSv1.2 | String196| *keyStoreFile* (security) | Client side certificate keystore to be used for encryption |  | File197| *keyStoreFormat* (security) | Keystore format to be used for payload encryption. Defaults to JKS if not set |  | String198| *keyStoreResource* (security) | Client side certificate keystore to be used for encryption. Is loaded by default from classpath, but you can prefix with classpath:, file:, or http: to load the resource from different systems. |  | String199| *needClientAuth* (security) | Configures whether the server needs client authentication when using SSL. | false | boolean200| *passphrase* (security) | Password setting to use in order to encrypt/decrypt payloads sent using SSH |  | String201| *securityConfiguration* (security) | Refers to a org.apache.camel.component.netty.http.NettyHttpSecurityConfiguration for configuring secure web resources. |  | NettyHttpSecurityConfiguration202| *securityProvider* (security) | Security provider to be used for payload encryption. Defaults to SunX509 if not set. |  | String203| *ssl* (security) | Setting to specify whether SSL encryption is applied to this endpoint | false | boolean204| *sslClientCertHeaders* (security) | When enabled and in SSL mode, then the Netty consumer will enrich the Camel Message with headers having information about the client certificate such as subject name, issuer name, serial number, and the valid date range. | false | boolean205| *sslContextParameters* (security) | To configure security using SSLContextParameters |  | SSLContextParameters206| *sslHandler* (security) | Reference to a class that could be used to return an SSL Handler |  | SslHandler207| *trustStoreFile* (security) | Server side certificate keystore to be used for encryption |  | File208| *trustStoreResource* (security) | Server side certificate keystore to be used for encryption. Is loaded by default from classpath, but you can prefix with classpath:, file:, or http: to load the resource from different systems. |  | String209| *useGlobalSslContextParameters* (security) | Enable usage of global SSL context parameters. | false | boolean210|===211// component options: END212 213// endpoint options: START214== Endpoint Options215 216The Netty HTTP endpoint is configured using URI syntax:217 218----219netty-http:protocol://host:port/path220----221 222with the following path and query parameters:223 224=== Path Parameters (4 parameters):225 226 227[width=""100%"",cols=""2,5,^1,2"",options=""header""]228|===229| Name | Description | Default | Type230| *protocol* | *Required* The protocol to use which is either http, https or proxy - a consumer only option. There are 2 enums and the value can be one of: http, https |  | String231| *host* | *Required* The local hostname such as localhost, or 0.0.0.0 when being a consumer. The remote HTTP server hostname when using producer. |  | String232| *port* | The host port number |  | int233| *path* | Resource path |  | String234|===235 236 237=== Query Parameters (80 parameters):238 239 240[width=""100%"",cols=""2,5,^1,2"",options=""header""]241|===242| Name | Description | Default | Type243| *bridgeEndpoint* (common) | If the option is true, the producer will ignore the Exchange.HTTP_URI header, and use the endpoint's URI for request. You may also set the throwExceptionOnFailure to be false to let the producer send all the fault response back. The consumer working in the bridge mode will skip the gzip compression and WWW URL form encoding (by adding the Exchange.SKIP_GZIP_ENCODING and Exchange.SKIP_WWW_FORM_URLENCODED headers to the consumed exchange). | false | boolean244| *disconnect* (common) | Whether or not to disconnect(close) from Netty Channel right after use. Can be used for both consumer and producer. | false | boolean245| *keepAlive* (common) | Setting to ensure socket is not closed due to inactivity | true | boolean246| *reuseAddress* (common) | Setting to facilitate socket multiplexing | true | boolean247| *reuseChannel* (common) | This option allows producers and consumers (in client mode) to reuse the same Netty Channel for the lifecycle of processing the Exchange. This is useful if you need to call a server multiple times in a Camel route and want to use the same network connection. When using this, the channel is not returned to the connection pool until the Exchange is done; or disconnected if the disconnect option is set to true. The reused Channel is stored on the Exchange as an exchange property with the key NettyConstants#NETTY_CHANNEL which allows you to obtain the channel during routing and use it as well. | false | boolean248| *sync* (common) | Setting to set endpoint as one-way or request-response | true | boolean249| *tcpNoDelay* (common) | Setting to improve TCP protocol performance | true | boolean250| *bridgeErrorHandler* (consumer) | Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions occurred while the consumer is trying to pickup incoming messages, or the likes, will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions, that will be logged at WARN or ERROR level and ignored. | false | boolean251| *matchOnUriPrefix* (consumer) | Whether or not Camel should try to find a target consumer by matching the URI prefix if no exact match is found. | false | boolean252| *muteException* (consumer) | If enabled and an Exchange failed processing on the consumer side the response's body won't contain the exception's stack trace. | false | boolean253| *send503whenSuspended* (consumer) | Whether to send back HTTP status code 503 when the consumer has been suspended. If the option is false then the Netty Acceptor is unbound when the consumer is suspended, so clients cannot connect anymore. | true | boolean254| *backlog* (consumer) | Allows to configure a backlog for netty consumer (server). Note the backlog is just a best effort depending on the OS. Setting this option to a value such as 200, 500 or 1000, tells the TCP stack how long the accept queue can be If this option is not configured, then the backlog depends on OS setting. |  | int255| *bossCount* (consumer) | When netty works on nio mode, it uses default bossCount parameter from Netty, which is 1. User can use this option to override the default bossCount from Netty | 1 | int256| *bossGroup* (consumer) | Set the BossGroup which could be used for handling the new connection of the server side across the NettyEndpoint |  | EventLoopGroup257| *chunkedMaxContentLength* (consumer) | Value in bytes the max content length per chunked frame received on the Netty HTTP server. | 1048576 | int258| *compression* (consumer) | Allow using gzip/deflate for compression on the Netty HTTP server if the client supports it from the HTTP headers. | false | boolean259| *disconnectOnNoReply* (consumer) | If sync is enabled then this option dictates NettyConsumer if it should disconnect where there is no reply to send back. | true | boolean260| *exceptionHandler* (consumer) | To let the consumer use a custom ExceptionHandler. Notice if the option bridgeErrorHandler is enabled then this option is not in use. By default the consumer will deal with exceptions, that will be logged at WARN or ERROR level and ignored. |  | ExceptionHandler261| *exchangePattern* (consumer) | Sets the exchange pattern when the consumer creates an exchange. There are 3 enums and the value can be one of: InOnly, InOut, InOptionalOut |  | ExchangePattern262| *httpMethodRestrict* (consumer) | To disable HTTP methods on the Netty HTTP consumer. You can specify multiple separated by comma. |  | String263| *logWarnOnBadRequest* (consumer) | Whether Netty HTTP server should log a WARN if decoding the HTTP request failed and a HTTP Status 400 (bad request) is returned. | true | boolean264| *mapHeaders* (consumer) | If this option is enabled, then during binding from Netty to Camel Message then the headers will be mapped as well (eg added as header to the Camel Message as well). You can turn off this option to disable this. The headers can still be accessed from the org.apache.camel.component.netty.http.NettyHttpMessage message with the method getHttpRequest() that returns the Netty HTTP request io.netty.handler.codec.http.HttpRequest instance. | true | boolean265| *maxHeaderSize* (consumer) | The maximum length of all headers. If the sum of the length of each header exceeds this value, a io.netty.handler.codec.TooLongFrameException will be raised. | 8192 | int266| *nettyServerBootstrapFactory* (consumer) | To use a custom NettyServerBootstrapFactory |  | NettyServerBootstrapFactory267| *nettySharedHttpServer* (consumer) | To use a shared Netty HTTP server. See Netty HTTP Server Example for more details. |  | NettySharedHttpServer268| *noReplyLogLevel* (consumer) | If sync is enabled this option dictates NettyConsumer which logging level to use when logging a there is no reply to send back. There are 6 enums and the value can be one of: TRACE, DEBUG, INFO, WARN, ERROR, OFF | WARN | LoggingLevel269| *serverClosedChannelException{zwsp}CaughtLogLevel* (consumer) | If the server (NettyConsumer) catches an java.nio.channels.ClosedChannelException then its logged using this logging level. This is used to avoid logging the closed channel exceptions, as clients can disconnect abruptly and then cause a flood of closed exceptions in the Netty server. There are 6 enums and the value can be one of: TRACE, DEBUG, INFO, WARN, ERROR, OFF | DEBUG | LoggingLevel270| *serverExceptionCaughtLogLevel* (consumer) | If the server (NettyConsumer) catches an exception then its logged using this logging level. There are 6 enums and the value can be one of: TRACE, DEBUG, INFO, WARN, ERROR, OFF | WARN | LoggingLevel271| *serverInitializerFactory* (consumer) | To use a custom ServerInitializerFactory |  | ServerInitializerFactory272| *traceEnabled* (consumer) | Specifies whether to enable HTTP TRACE for this Netty HTTP consumer. By default TRACE is turned off. | false | boolean273| *urlDecodeHeaders* (consumer) | If this option is enabled, then during binding from Netty to Camel Message then the header values will be URL decoded (eg %20 will be a space character. Notice this option is used by the default org.apache.camel.component.netty.http.NettyHttpBinding and therefore if you implement a custom org.apache.camel.component.netty.http.NettyHttpBinding then you would need to decode the headers accordingly to this option. | false | boolean274| *usingExecutorService* (consumer) | Whether to use ordered thread pool, to ensure events are processed orderly on the same channel. | true | boolean275| *connectTimeout* (producer) | Time to wait for a socket connection to be available. Value is in milliseconds. | 10000 | int276| *cookieHandler* (producer) | Configure a cookie handler to maintain a HTTP session |  | CookieHandler277| *lazyStartProducer* (producer) | Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel's routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing. | false | boolean278| *requestTimeout* (producer) | Allows to use a timeout for the Netty producer when calling a remote server. By default no timeout is in use. The value is in milli seconds, so eg 30000 is 30 seconds. The requestTimeout is using Netty's ReadTimeoutHandler to trigger the timeout. |  | long279| *throwExceptionOnFailure* (producer) | Option to disable throwing the HttpOperationFailedException in case of failed responses from the remote server. This allows you to get all responses regardless of the HTTP status code. | true | boolean280| *clientInitializerFactory* (producer) | To use a custom ClientInitializerFactory |  | ClientInitializerFactory281| *lazyChannelCreation* (producer) | Channels can be lazily created to avoid exceptions, if the remote server is not up and running when the Camel producer is started. | true | boolean282| *okStatusCodeRange* (producer) | The status codes which are considered a success response. The values are inclusive. Multiple ranges can be defined, separated by comma, e.g. 200-204,209,301-304. Each range must be a single number or from-to with the dash included. The default range is 200-299 | 200-299 | String283| *producerPoolEnabled* (producer) | Whether producer pool is enabled or not. Important: If you turn this off then a single shared connection is used for the producer, also if you are doing request/reply. That means there is a potential issue with interleaved responses if replies comes back out-of-order. Therefore you need to have a correlation id in both the request and reply messages so you can properly correlate the replies to the Camel callback that is responsible for continue processing the message in Camel. To do this you need to implement NettyCamelStateCorrelationManager as correlation manager and configure it via the correlationManager option. See also the correlationManager option for more details. | true | boolean284| *producerPoolMaxActive* (producer) | Sets the cap on the number of objects that can be allocated by the pool (checked out to clients, or idle awaiting checkout) at a given time. Use a negative value for no limit. | -1 | int285| *producerPoolMaxIdle* (producer) | Sets the cap on the number of idle instances in the pool. | 100 | int286| *producerPoolMinEvictableIdle* (producer) | Sets the minimum amount of time (value in millis) an object may sit idle in the pool before it is eligible for eviction by the idle object evictor. | 300000 | long287| *producerPoolMinIdle* (producer) | Sets the minimum number of instances allowed in the producer pool before the evictor thread (if active) spawns new objects. |  | int288| *useRelativePath* (producer) | Sets whether to use a relative path in HTTP requests. | true | boolean289| *hostnameVerification* ( security) | To enable/disable hostname verification on SSLEngine | false | boolean290| *allowSerializedHeaders* (advanced) | Only used for TCP when transferExchange is true. When set to true, serializable objects in headers and properties will be added to the exchange. Otherwise Camel will exclude any non-serializable objects and log it at WARN level. | false | boolean291| *channelGroup* (advanced) | To use a explicit ChannelGroup. |  | ChannelGroup292| *configuration* (advanced) | To use a custom configured NettyHttpConfiguration for configuring this endpoint. |  | NettyHttpConfiguration293| *disableStreamCache* (advanced) | Determines whether or not the raw input stream from Netty HttpRequest#getContent() or HttpResponset#getContent() is cached or not (Camel will read the stream into a in light-weight memory based Stream caching) cache. By default Camel will cache the Netty input stream to support reading it multiple times to ensure it Camel can retrieve all data from the stream. However you can set this option to true when you for example need to access the raw stream, such as streaming it directly to a file or other persistent store. Mind that if you enable this option, then you cannot read the Netty stream multiple times out of the box, and you would need manually to reset the reader index on the Netty raw stream. Also Netty will auto-close the Netty stream when the Netty HTTP server/HTTP client is done processing, which means that if the asynchronous routing engine is in use then any asynchronous thread that may continue routing the org.apache.camel.Exchange may not be able to read the Netty stream, because Netty has closed it. | false | boolean294| *headerFilterStrategy* (advanced) | To use a custom org.apache.camel.spi.HeaderFilterStrategy to filter headers. |  | HeaderFilterStrategy295| *nativeTransport* (advanced) | Whether to use native transport instead of NIO. Native transport takes advantage of the host operating system and is only supported on some platforms. You need to add the netty JAR for the host operating system you are using. See more details at: \\http://netty.io/wiki/native-transports.html | false | boolean296| *nettyHttpBinding* (advanced) | To use a custom org.apache.camel.component.netty.http.NettyHttpBinding for binding to/from Netty and Camel Message API. |  | NettyHttpBinding297| *options* (advanced) | Allows to configure additional netty options using option. as prefix. For example option.child.keepAlive=false to set the netty option child.keepAlive=false. See the Netty documentation for possible options that can be used. |  | Map298| *receiveBufferSize* (advanced) | The TCP/UDP buffer sizes to be used during inbound communication. Size is bytes. | 65536 | int299| *receiveBufferSizePredictor* (advanced) | Configures the buffer size predictor. See details at Jetty documentation and this mail thread. |  | int300| *sendBufferSize* (advanced) | The TCP/UDP buffer sizes to be used during outbound communication. Size is bytes. | 65536 | int301| *synchronous* (advanced) | Sets whether synchronous processing should be strictly used | false | boolean302| *transferException* (advanced) | If enabled and an Exchange failed processing on the consumer side, and if the caused Exception was send back serialized in the response as a application/x-java-serialized-object content type. On the producer side the exception will be deserialized and thrown as is, instead of the HttpOperationFailedException. The caused exception is required to be serialized. This is by default turned off. If you enable this then be aware that Java will deserialize the incoming data from the request to Java and that can be a potential security risk. | false | boolean303| *transferExchange* (advanced) | Only used for TCP. You can transfer the exchange over the wire instead of just the body. The following fields are transferred: In body, Out body, fault body, In headers, Out headers, fault headers, exchange properties, exchange exception. This requires that the objects are serializable. Camel will exclude any non-serializable objects and log it at WARN level. | false | boolean304| *workerCount* (advanced) | When netty works on nio mode, it uses default workerCount parameter from Netty (which is cpu_core_threads x 2). User can use this option to override the default workerCount from Netty. |  | int305| *workerGroup* (advanced) | To use a explicit EventLoopGroup as the boss thread pool. For example to share a thread pool with multiple consumers or producers. By default each consumer or producer has their own worker pool with 2 x cpu count core threads. |  | EventLoopGroup306| *decoders* (codec) | A list of decoders to be used. You can use a String which have values separated by comma, and have the values be looked up in the Registry. Just remember to prefix the value with # so Camel knows it should lookup. |  | List307| *encoders* (codec) | A list of encoders to be used. You can use a String which have values separated by comma, and have the values be looked up in the Registry. Just remember to prefix the value with # so Camel knows it should lookup. |  | List308| *enabledProtocols* (security) | Which protocols to enable when using SSL | TLSv1,TLSv1.1,TLSv1.2 | String309| *keyStoreFile* (security) | Client side certificate keystore to be used for encryption |  | File310| *keyStoreFormat* (security) | Keystore format to be used for payload encryption. Defaults to JKS if not set |  | String311| *keyStoreResource* (security) | Client side certificate keystore to be used for encryption. Is loaded by default from classpath, but you can prefix with classpath:, file:, or http: to load the resource from different systems. |  | String312| *needClientAuth* (security) | Configures whether the server needs client authentication when using SSL. | false | boolean313| *passphrase* (security) | Password setting to use in order to encrypt/decrypt payloads sent using SSH |  | String314| *securityConfiguration* (security) | Refers to a org.apache.camel.component.netty.http.NettyHttpSecurityConfiguration for configuring secure web resources. |  | NettyHttpSecurityConfiguration315| *securityOptions* (security) | To configure NettyHttpSecurityConfiguration using key/value pairs from the map |  | Map316| *securityProvider* (security) | Security provider to be used for payload encryption. Defaults to SunX509 if not set. |  | String317| *ssl* (security) | Setting to specify whether SSL encryption is applied to this endpoint | false | boolean318| *sslClientCertHeaders* (security) | When enabled and in SSL mode, then the Netty consumer will enrich the Camel Message with headers having information about the client certificate such as subject name, issuer name, serial number, and the valid date range. | false | boolean319| *sslContextParameters* (security) | To configure security using SSLContextParameters |  | SSLContextParameters320| *sslHandler* (security) | Reference to a class that could be used to return an SSL Handler |  | SslHandler321| *trustStoreFile* (security) | Server side certificate keystore to be used for encryption |  | File322| *trustStoreResource* (security) | Server side certificate keystore to be used for encryption. Is loaded by default from classpath, but you can prefix with classpath:, file:, or http: to load the resource from different systems. |  | String323|===324// endpoint options: END325 326 327 328== Message Headers329 330The following headers can be used on the producer to control the HTTP331request.332 333[width=""100%"",cols=""10%,10%,80%"",options=""header"",]334|=======================================================================335|Name |Type |Description336 337|`CamelHttpMethod` |`String` |Allow to control what HTTP method to use such as GET, POST, TRACE etc.338The type can also be a `io.netty.handler.codec.http.HttpMethod`339instance.340 341|`CamelHttpQuery` |`String` |Allows to provide URI query parameters as a `String` value that342overrides the endpoint configuration. Separate multiple parameters using343the & sign. For example: `foo=bar&beer=yes`.344 345|`CamelHttpPath` |`String` |Allows to provide URI context-path and query parameters as a `String`346value that overrides the endpoint configuration. This allows to reuse347the same producer for calling same remote http server, but using a348dynamic context-path and query parameters.349 350|`Content-Type` |`String` |To set the content-type of the HTTP body. For example:351`text/plain; charset=""UTF-8""`.352 353|`CamelHttpResponseCode` |`int` |Allows to set the HTTP Status code to use. By default 200 is used for354success, and 500 for failure.355|=======================================================================356 357The following headers is provided as meta-data when a route starts from358an Netty HTTP endpoint:359 360The description in the table takes offset in a route having:361`from(""netty-http:http:0.0.0.0:8080/myapp"")...`362 363[width=""100%"",cols=""10%,10%,80%"",options=""header"",]364|=======================================================================365|Name |Type |Description366 367|`CamelHttpMethod` |`String` |The HTTP method used, such as GET, POST, TRACE etc.368 369|`CamelHttpUrl` |`String` |The URL including protocol, host and port, etc: 370`\\http://0.0.0.0:8080/myapp`371 372|`CamelHttpUri` |`String` |The URI without protocol, host and port, etc:373`/myapp`374 375|`CamelHttpQuery` |`String` |Any query parameters, such as `foo=bar&beer=yes`376 377|`CamelHttpRawQuery` |`String` |Any query parameters, such as `foo=bar&beer=yes`. Stored in the raw378form, as they arrived to the consumer (i.e. before URL decoding).379 380|`CamelHttpPath` |`String` |Additional context-path. This value is empty if the client called the381context-path `/myapp`. If the client calls `/myapp/mystuff`, then this382header value is `/mystuff`. In other words its the value after the383context-path configured on the route endpoint.384 385|`CamelHttpCharacterEncoding` |`String` |The charset from the content-type header.386 387|`CamelHttpAuthentication` |`String` |If the user was authenticated using HTTP Basic then this header is added388with the value `Basic`.389 390|`Content-Type` |`String` |The content type if provided. For example:391`text/plain; charset=""UTF-8""`.392|=======================================================================393 394== Access to Netty types395 396This component uses the397`org.apache.camel.component.netty.http.NettyHttpMessage` as the message398implementation on the Exchange. This allows end399users to get access to the original Netty request/response instances if400needed, as shown below. Mind that the original response may not be401accessible at all times.402 403[source,java]404----------------------------------------------------------------------------------------------------------405io.netty.handler.codec.http.HttpRequest request = exchange.getIn(NettyHttpMessage.class).getHttpRequest();406----------------------------------------------------------------------------------------------------------407 408== Examples409 410In the route below we use Netty HTTP as a HTTP server, which returns411back a hardcoded ""Bye World"" message.412 413[source,java]414-----------------------------------------------415    from(""netty-http:http://0.0.0.0:8080/foo"")416      .transform().constant(""Bye World"");417-----------------------------------------------418 419And we can call this HTTP server using Camel also, with the420ProducerTemplate as shown below:421 422[source,java]423------------------------------------------------------------------------------------------------------------424    String out = template.requestBody(""netty-http:http://0.0.0.0:8080/foo"", ""Hello World"", String.class);425    System.out.println(out);426------------------------------------------------------------------------------------------------------------427 428And we get back ""Bye World"" as the output.429 430=== How do I let Netty match wildcards431 432By default Netty HTTP will only match on exact uri's. But you can433instruct Netty to match prefixes. For example434 435[source,java]436-----------------------------------------------------------437from(""netty-http:http://0.0.0.0:8123/foo"").to(""mock:foo"");438-----------------------------------------------------------439 440In the route above Netty HTTP will only match if the uri is an exact441match, so it will match if you enter +442 `\\http://0.0.0.0:8123/foo` but not match if you do443`\\http://0.0.0.0:8123/foo/bar`.444 445So if you want to enable wildcard matching you do as follows:446 447[source,java]448---------------------------------------------------------------------------------449from(""netty-http:http://0.0.0.0:8123/foo?matchOnUriPrefix=true"").to(""mock:foo"");450---------------------------------------------------------------------------------451 452So now Netty matches any endpoints with starts with `foo`.453 454To match *any* endpoint you can do:455 456[source,java]457-----------------------------------------------------------------------------458from(""netty-http:http://0.0.0.0:8123?matchOnUriPrefix=true"").to(""mock:foo"");459-----------------------------------------------------------------------------460 461=== Using multiple routes with same port462 463In the same CamelContext you can have multiple464routes from Netty HTTP that shares the same port (eg a465`io.netty.bootstrap.ServerBootstrap` instance). Doing this requires a466number of bootstrap options to be identical in the routes, as the routes467will share the same `io.netty.bootstrap.ServerBootstrap` instance. The468instance will be configured with the options from the first route469created.470 471The options the routes must be identical configured is all the options472defined in the473`org.apache.camel.component.netty.NettyServerBootstrapConfiguration`474configuration class. If you have configured another route with different475options, Camel will throw an exception on startup, indicating the476options is not identical. To mitigate this ensure all options is477identical.478 479Here is an example with two routes that share the same port.480 481*Two routes sharing the same port*482 483[source,java]484-----------------------------------------------485from(""netty-http:http://0.0.0.0:{{port}}/foo"")486  .to(""mock:foo"")487  .transform().constant(""Bye World"");488 489from(""netty-http:http://0.0.0.0:{{port}}/bar"")490  .to(""mock:bar"")491  .transform().constant(""Bye Camel"");492-----------------------------------------------493 494And here is an example of a mis configured 2nd route that do not have495identical496`org.apache.camel.component.netty.NettyServerBootstrapConfiguration`497option as the 1st route. This will cause Camel to fail on startup.498 499*Two routes sharing the same port, but the 2nd route is misconfigured500and will fail on starting*501 502[source,java]503--------------------------------------------------------------------------------------504from(""netty-http:http://0.0.0.0:{{port}}/foo"")505  .to(""mock:foo"")506  .transform().constant(""Bye World"");507 508// we cannot have a 2nd route on same port with SSL enabled, when the 1st route is NOT509from(""netty-http:http://0.0.0.0:{{port}}/bar?ssl=true"")510  .to(""mock:bar"")511  .transform().constant(""Bye Camel"");512--------------------------------------------------------------------------------------513 514=== Reusing same server bootstrap configuration with multiple routes515 516By configuring the common server bootstrap option in an single instance517of a518`org.apache.camel.component.netty.NettyServerBootstrapConfiguration`519type, we can use the `bootstrapConfiguration` option on the Netty HTTP520consumers to refer and reuse the same options across all consumers.521 522[source,xml]523-----------------------------------------------------------------------------------------------------------------524<bean id=""nettyHttpBootstrapOptions"" class=""org.apache.camel.component.netty.NettyServerBootstrapConfiguration"">525  <property name=""backlog"" value=""200""/>526  <property name=""connectionTimeout"" value=""20000""/>527  <property name=""workerCount"" value=""16""/>528</bean>529-----------------------------------------------------------------------------------------------------------------530 531And in the routes you refer to this option as shown below532 533[source,xml]534----------------------------------------------------------------------------------------------------------535<route>536  <from uri=""netty-http:http://0.0.0.0:{{port}}/foo?bootstrapConfiguration=#nettyHttpBootstrapOptions""/>537  ...538</route>539 540<route>541  <from uri=""netty-http:http://0.0.0.0:{{port}}/bar?bootstrapConfiguration=#nettyHttpBootstrapOptions""/>542  ...543</route>544 545<route>546  <from uri=""netty-http:http://0.0.0.0:{{port}}/beer?bootstrapConfiguration=#nettyHttpBootstrapOptions""/>547  ...548</route>549----------------------------------------------------------------------------------------------------------550 551=== Reusing same server bootstrap configuration with multiple routes across multiple bundles in OSGi container552 553See the Netty HTTP Server Example554for more details and example how to do that.555 556=== Implementing a reverse proxy557 558Netty HTTP component can act as a reverse proxy, in that case559`Exchange.HTTP_SCHEME`, `Exchange.HTTP_HOST` and560`Exchange.HTTP_PORT` headers are populated from the absolute561URL received on the request line of the HTTP request.562 563Here's an example of a HTTP proxy that simply transforms the response564from the origin server to uppercase.565 566[source,java]567------------------------------------------------------------------------------------------568from(""netty-http:proxy://0.0.0.0:8080"")569    .toD(""netty-http:""570        + ""${headers."" + Exchange.HTTP_SCHEME + ""}://""571        + ""${headers."" + Exchange.HTTP_HOST + ""}:""572        + ""${headers."" + Exchange.HTTP_PORT + ""}"")573    .process(this::processResponse);574 575void processResponse(final Exchange exchange) {576    final NettyHttpMessage message = exchange.getIn(NettyHttpMessage.class);577    final FullHttpResponse response = message.getHttpResponse();578 579    final ByteBuf buf = response.content();580    final String string = buf.toString(StandardCharsets.UTF_8);581 582    buf.resetWriterIndex();583    ByteBufUtil.writeUtf8(buf, string.toUpperCase(Locale.US));584}585------------------------------------------------------------------------------------------586 587== Using HTTP Basic Authentication588 589The Netty HTTP consumer supports HTTP basic authentication by specifying590the security realm name to use, as shown below591 592[source,java]593------------------------------------------------------------------------------------------594<route>595   <from uri=""netty-http:http://0.0.0.0:{{port}}/foo?securityConfiguration.realm=karaf""/>596   ...597</route>598------------------------------------------------------------------------------------------599 600The realm name is mandatory to enable basic authentication. By default601the JAAS based authenticator is used, which will use the realm name602specified (karaf in the example above) and use the JAAS realm and the603JAAS \\{\\{LoginModule}}s of this realm for authentication.604 605End user of Apache Karaf / ServiceMix has a karaf realm out of the box,606and hence why the example above would work out of the box in these607containers.608 609=== Specifying ACL on web resources610 611The `org.apache.camel.component.netty.http.SecurityConstraint` allows612to define constrains on web resources. And the613`org.apache.camel.component.netty.http.SecurityConstraintMapping` is614provided out of the box, allowing to easily define inclusions and615exclusions with roles.616 617For example as shown below in the XML DSL, we define the constraint618bean:619 620[source,xml]621-------------------------------------------------------------------------------------------------622  <bean id=""constraint"" class=""org.apache.camel.component.netty.http.SecurityConstraintMapping"">623    <!-- inclusions defines url -> roles restrictions -->624    <!-- a * should be used for any role accepted (or even no roles) -->625    <property name=""inclusions"">626      <map>627        <entry key=""/*"" value=""*""/>628        <entry key=""/admin/*"" value=""admin""/>629        <entry key=""/guest/*"" value=""admin,guest""/>630      </map>631    </property>632    <!-- exclusions is used to define public urls, which requires no authentication -->633    <property name=""exclusions"">634      <set>635        <value>/public/*</value>636      </set>637    </property>638  </bean>639-------------------------------------------------------------------------------------------------640 641The constraint above is define so that642 643* access to /* is restricted and any roles is accepted (also if user has644no roles)645* access to /admin/* requires the admin role646* access to /guest/* requires the admin or guest role647* access to /public/* is an exclusion which means no authentication is648needed, and is therefore public for everyone without logging in649 650To use this constraint we just need to refer to the bean id as shown651below:652 653[source,xml]654-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------655<route>656   <from uri=""netty-http:http://0.0.0.0:{{port}}/foo?matchOnUriPrefix=true&amp;securityConfiguration.realm=karaf&amp;securityConfiguration.securityConstraint=#constraint""/>657   ...658</route>659-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------660 661 662include::{page-component-version}@camel-spring-boot::page$netty-http-starter.adoc[]663",12292,True,2518.27649669739,median664"////665/**666 *667 * Licensed to the Apache Software Foundation (ASF) under one668 * or more contributor license agreements.  See the NOTICE file669 * distributed with this work for additional information670 * regarding copyright ownership.  The ASF licenses this file671 * to you under the Apache License, Version 2.0 (the672 * ""License""); you may not use this file except in compliance673 * with the License.  You may obtain a copy of the License at674 *675 *     http://www.apache.org/licenses/LICENSE-2.0676 *677 * Unless required by applicable law or agreed to in writing, software678 * distributed under the License is distributed on an ""AS IS"" BASIS,679 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.680 * See the License for the specific language governing permissions and681 * limitations under the License.682 */683////684 685[[upgrading]]686= Upgrading687:doctype: book688:numbered:689:toc: left690:icons: font691:experimental:692 693You cannot skip major versions when upgrading. If you are upgrading from version 0.98.x to 2.x, you must first go from 0.98.x to 1.2.x and then go from 1.2.x to 2.x.694 695Review <<configuration>>, in particular <<hadoop>>. Familiarize yourself with <<hbase_supported_tested_definitions>>.696 697[[hbase.versioning]]698== HBase version number and compatibility699 700[[hbase.versioning.post10]]701=== Aspirational Semantic Versioning702 703Starting with the 1.0.0 release, HBase is working towards link:http://semver.org/[Semantic Versioning] for its release versioning. In summary:704 705.Given a version number MAJOR.MINOR.PATCH, increment the:706* MAJOR version when you make incompatible API changes,707* MINOR version when you add functionality in a backwards-compatible manner, and708* PATCH version when you make backwards-compatible bug fixes.709* Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.710 711[[hbase.versioning.compat]]712.Compatibility Dimensions713In addition to the usual API versioning considerations HBase has other compatibility dimensions that we need to consider.714 715.Client-Server wire protocol compatibility716* Allows updating client and server out of sync.717* We could only allow upgrading the server first. I.e. the server would be backward compatible to an old client, that way new APIs are OK.718* Example: A user should be able to use an old client to connect to an upgraded cluster.719 720.Server-Server protocol compatibility721* Servers of different versions can co-exist in the same cluster.722* The wire protocol between servers is compatible.723* Workers for distributed tasks, such as replication and log splitting, can co-exist in the same cluster.724* Dependent protocols (such as using ZK for coordination) will also not be changed.725* Example: A user can perform a rolling upgrade.726 727.File format compatibility728* Support file formats backward and forward compatible729* Example: File, ZK encoding, directory layout is upgraded automatically as part of an HBase upgrade. User can downgrade to the older version and everything will continue to work.730 731.Client API compatibility732* Allow changing or removing existing client APIs.733* An API needs to be deprecated for a major version before we will change/remove it.734* APIs available in a patch version will be available in all later patch versions. However, new APIs may be added which will not be available in earlier patch versions.735* New APIs introduced in a patch version will only be added in a source compatible way footnote:[See 'Source Compatibility' https://blogs.oracle.com/darcy/entry/kinds_of_compatibility]: i.e. code that implements public APIs will continue to compile.736** Example: A user using a newly deprecated API does not need to modify application code with HBase API calls until the next major version.737*738 739.Client Binary compatibility740* Client code written to APIs available in a given patch release can run unchanged (no recompilation needed) against the new jars of later patch versions.741* Client code written to APIs available in a given patch release might not run against the old jars from an earlier patch version.742** Example: Old compiled client code will work unchanged with the new jars.743* If a Client implements an HBase Interface, a recompile MAY be required upgrading to a newer minor version (See release notes744for warning about incompatible changes). All effort will be made to provide a default implementation so this case should not arise.745 746.Server-Side Limited API compatibility (taken from Hadoop)747* Internal APIs are marked as Stable, Evolving, or Unstable748* This implies binary compatibility for coprocessors and plugins (pluggable classes, including replication) as long as these are only using marked interfaces/classes.749* Example: Old compiled Coprocessor, Filter, or Plugin code will work unchanged with the new jars.750 751.Dependency Compatibility752* An upgrade of HBase will not require an incompatible upgrade of a dependent project, except for Apache Hadoop.753* An upgrade of HBase will not require an incompatible upgrade of the Java runtime.754* Example: Upgrading HBase to a version that supports _Dependency Compatibility_ won't require that you upgrade your Apache ZooKeeper service.755* Example: If your current version of HBase supported running on JDK 8, then an upgrade to a version that supports _Dependency Compatibility_ will also run on JDK 8.756 757.Hadoop Versions758[TIP]759====760Previously, we tried to maintain dependency compatibility for the underly Hadoop service but over the last few years this has proven untenable. While the HBase project attempts to maintain support for older versions of Hadoop, we drop the ""supported"" designator for minor versions that fail to continue to see releases. Additionally, the Hadoop project has its own set of compatibility guidelines, which means in some cases having to update to a newer supported minor release might break some of our compatibility promises.761====762 763.Operational Compatibility764* Metric changes765* Behavioral changes of services766* JMX APIs exposed via the `/jmx/` endpoint767 768.Summary769* A patch upgrade is a drop-in replacement. Any change that is not Java binary and source compatible would not be allowed.footnote:[See http://docs.oracle.com/javase/specs/jls/se7/html/jls-13.html.] Downgrading versions within patch releases may not be compatible.770 771* A minor upgrade requires no application/client code modification. Ideally it would be a drop-in replacement but client code, coprocessors, filters, etc might have to be recompiled if new jars are used.772 773* A major upgrade allows the HBase community to make breaking changes.774 775.Compatibility Matrix footnote:[Note that this indicates what could break, not that it will break. We will/should add specifics in our release notes.]776[cols=""1,1,1,1""]777|===778| | Major | Minor | Patch779|Client-Server wire Compatibility|  N |Y |Y780|Server-Server Compatibility |N |Y |Y781|File Format Compatibility | N footnote:[comp_matrix_offline_upgrade_note,Running an offline upgrade tool without downgrade might be needed. We will typically only support migrating data from major version X to major version X+1.] | Y |Y782|Client API Compatibility  | N | Y |Y783|Client Binary Compatibility | N | N |Y7844+|Server-Side Limited API Compatibility785>| Stable | N | Y | Y786>| Evolving | N |N |Y787>| Unstable | N |N |N788|Dependency Compatibility | N |Y |Y789|Operational Compatibility | N |N |Y790|===791 792[[hbase.client.api.surface]]793==== HBase API Surface794 795HBase has a lot of API points, but for the compatibility matrix above, we differentiate between Client API, Limited Private API, and Private API. HBase uses link:https://yetus.apache.org/documentation/0.5.0/interface-classification/[Apache Yetus Audience Annotations] to guide downstream expectations for stability.796 797* InterfaceAudience (link:https://yetus.apache.org/documentation/0.5.0/audience-annotations-apidocs/org/apache/yetus/audience/InterfaceAudience.html[javadocs]): captures the intended audience, possible values include:798  - Public: safe for end users and external projects799  - LimitedPrivate: used for internals we expect to be pluggable, such as coprocessors800  - Private: strictly for use within HBase itself801Classes which are defined as `IA.Private` may be used as parameters or return values for interfaces which are declared `IA.LimitedPrivate`. Treat the `IA.Private` object as opaque; do not try to access its methods or fields directly.802* InterfaceStability (link:https://yetus.apache.org/documentation/0.5.0/audience-annotations-apidocs/org/apache/yetus/audience/InterfaceStability.html[javadocs]): describes what types of interface changes are permitted. Possible values include:803  - Stable: the interface is fixed and is not expected to change804  - Evolving: the interface may change in future minor verisons805  - Unstable: the interface may change at any time806 807Please keep in mind the following interactions between the `InterfaceAudience` and `InterfaceStability` annotations within the HBase project:808 809* `IA.Public` classes are inherently stable and adhere to our stability guarantees relating to the type of upgrade (major, minor, or patch).810* `IA.LimitedPrivate` classes should always be annotated with one of the given `InterfaceStability` values. If they are not, you should presume they are `IS.Unstable`.811* `IA.Private` classes should be considered implicitly unstable, with no guarantee of stability between releases.812 813[[hbase.client.api]]814HBase Client API::815  HBase Client API consists of all the classes or methods that are marked with InterfaceAudience.Public interface. All main classes in hbase-client and dependent modules have either InterfaceAudience.Public, InterfaceAudience.LimitedPrivate, or InterfaceAudience.Private marker. Not all classes in other modules (hbase-server, etc) have the marker. If a class is not annotated with one of these, it is assumed to be a InterfaceAudience.Private class.816 817[[hbase.limitetprivate.api]]818HBase LimitedPrivate API::819  LimitedPrivate annotation comes with a set of target consumers for the interfaces. Those consumers are coprocessors, phoenix, replication endpoint implementations or similar. At this point, HBase only guarantees source and binary compatibility for these interfaces between patch versions.820 821[[hbase.private.api]]822HBase Private API::823  All classes annotated with InterfaceAudience.Private or all classes that do not have the annotation are for HBase internal use only. The interfaces and method signatures can change at any point in time. If you are relying on a particular interface that is marked Private, you should open a jira to propose changing the interface to be Public or LimitedPrivate, or an interface exposed for this purpose.824 825[[hbase.binary.compatibility]]826.Binary Compatibility827When we say two HBase versions are compatible, we mean that the versions are wire and binary compatible. Compatible HBase versions means that clients can talk to compatible but differently versioned servers. It means too that you can just swap out the jars of one version and replace them with the jars of another, compatible version and all will just work. Unless otherwise specified, HBase point versions are (mostly) binary compatible. You can safely do rolling upgrades between binary compatible versions; i.e. across maintenance releases: e.g. from 1.2.4 to 1.2.6. See link:[Does compatibility between versions also mean binary compatibility?] discussion on the HBase dev mailing list.828 829[[hbase.rolling.upgrade]]830=== Rolling Upgrades831 832A rolling upgrade is the process by which you update the servers in your cluster a server at a time. You can rolling upgrade across HBase versions if they are binary or wire compatible. See <<hbase.rolling.restart>> for more on what this means. Coarsely, a rolling upgrade is a graceful stop each server, update the software, and then restart. You do this for each server in the cluster. Usually you upgrade the Master first and then the RegionServers. See <<rolling>> for tools that can help use the rolling upgrade process.833 834For example, in the below, HBase was symlinked to the actual HBase install. On upgrade, before running a rolling restart over the cluster, we changed the symlink to point at the new HBase software version and then ran835 836[source,bash]837----838$ HADOOP_HOME=~/hadoop-2.6.0-CRC-SNAPSHOT ~/hbase/bin/rolling-restart.sh --config ~/conf_hbase839----840 841The rolling-restart script will first gracefully stop and restart the master, and then each of the RegionServers in turn. Because the symlink was changed, on restart the server will come up using the new HBase version. Check logs for errors as the rolling upgrade proceeds.842 843[[hbase.rolling.restart]]844.Rolling Upgrade Between Versions that are Binary/Wire Compatible845Unless otherwise specified, HBase minor versions are binary compatible. You can do a <<hbase.rolling.upgrade>> between HBase point versions. For example, you can go to 1.2.4 from 1.2.6 by doing a rolling upgrade across the cluster replacing the 1.2.4 binary with a 1.2.6 binary.846 847In the minor version-particular sections below, we call out where the versions are wire/protocol compatible and in this case, it is also possible to do a <<hbase.rolling.upgrade>>.848 849== Rollback850 851Sometimes things don't go as planned when attempting an upgrade. This section explains how to perform a _rollback_ to an earlier HBase release. Note that this should only be needed between Major and some Minor releases. You should always be able to _downgrade_ between HBase Patch releases within the same Minor version. These instructions may require you to take steps before you start the upgrade process, so be sure to read through this section beforehand.852 853=== Caveats854 855.Rollback vs Downgrade856This section describes how to perform a _rollback_ on an upgrade between HBase minor and major versions. In this document, rollback refers to the process of taking an upgraded cluster and restoring it to the old version _while losing all changes that have occurred since upgrade_. By contrast, a cluster _downgrade_ would restore an upgraded cluster to the old version while maintaining any data written since the upgrade. We currently only offer instructions to rollback HBase clusters. Further, rollback only works when these instructions are followed prior to performing the upgrade.857 858When these instructions talk about rollback vs downgrade of prerequisite cluster services (i.e. HDFS), you should treat leaving the service version the same as a degenerate case of downgrade.859 860.Replication861Unless you are doing an all-service rollback, the HBase cluster will lose any configured peers for HBase replication. If your cluster is configured for HBase replication, then prior to following these instructions you should document all replication peers. After performing the rollback you should then add each documented peer back to the cluster. For more information on enabling HBase replication, listing peers, and adding a peer see <<hbase.replication.management>>. Note also that data written to the cluster since the upgrade may or may not have already been replicated to any peers. Determining which, if any, peers have seen replication data as well as rolling back the data in those peers is out of the scope of this guide.862 863.Data Locality864Unless you are doing an all-service rollback, going through a rollback procedure will likely destroy all locality for Region Servers. You should expect degraded performance until after the cluster has had time to go through compactions to restore data locality. Optionally, you can force a compaction to speed this process up at the cost of generating cluster load.865 866.Configurable Locations867The instructions below assume default locations for the HBase data directory and the HBase znode. Both of these locations are configurable and you should verify the value used in your cluster before proceeding. In the event that you have a different value, just replace the default with the one found in your configuration868* HBase data directory is configured via the key 'hbase.rootdir' and has a default value of '/hbase'.869* HBase znode is configured via the key 'zookeeper.znode.parent' and has a default value of '/hbase'.870 871=== All service rollback872 873If you will be performing a rollback of both the HDFS and ZooKeeper services, then HBase's data will be rolled back in the process.874 875.Requirements876 877* Ability to rollback HDFS and ZooKeeper878 879.Before upgrade880No additional steps are needed pre-upgrade. As an extra precautionary measure, you may wish to use distcp to back up the HBase data off of the cluster to be upgraded. To do so, follow the steps in the 'Before upgrade' section of 'Rollback after HDFS downgrade' but copy to another HDFS instance instead of within the same instance.881 882.Performing a rollback883 884. Stop HBase885. Perform a rollback for HDFS and ZooKeeper (HBase should remain stopped)886. Change the installed version of HBase to the previous version887. Start HBase888. Verify HBase contents—use the HBase shell to list tables and scan some known values.889 890=== Rollback after HDFS rollback and ZooKeeper downgrade891 892If you will be rolling back HDFS but going through a ZooKeeper downgrade, then HBase will be in an inconsistent state. You must ensure the cluster is not started until you complete this process.893 894.Requirements895 896* Ability to rollback HDFS897* Ability to downgrade ZooKeeper898 899.Before upgrade900No additional steps are needed pre-upgrade. As an extra precautionary measure, you may wish to use distcp to back up the HBase data off of the cluster to be upgraded. To do so, follow the steps in the 'Before upgrade' section of 'Rollback after HDFS downgrade' but copy to another HDFS instance instead of within the same instance.901 902.Performing a rollback903 904. Stop HBase905. Perform a rollback for HDFS and a downgrade for ZooKeeper (HBase should remain stopped)906. Change the installed version of HBase to the previous version907. Clean out ZooKeeper information related to HBase. WARNING: This step will permanently destroy all replication peers. Please see the section on HBase Replication under Caveats for more information.908+909.Clean HBase information out of ZooKeeper910[source,bash]911----912[hpnewton@gateway_node.example.com ~]$ zookeeper-client -server zookeeper1.example.com:2181,zookeeper2.example.com:2181,zookeeper3.example.com:2181913Welcome to ZooKeeper!914JLine support is disabled915rmr /hbase916quit917Quitting...918----919. Start HBase920. Verify HBase contents—use the HBase shell to list tables and scan some known values.921 922=== Rollback after HDFS downgrade923 924If you will be performing an HDFS downgrade, then you'll need to follow these instructions regardless of whether ZooKeeper goes through rollback, downgrade, or reinstallation.925 926.Requirements927 928* Ability to downgrade HDFS929* Pre-upgrade cluster must be able to run MapReduce jobs930* HDFS super user access931* Sufficient space in HDFS for at least two copies of the HBase data directory932 933.Before upgrade934Before beginning the upgrade process, you must take a complete backup of HBase's backing data. The following instructions cover backing up the data within the current HDFS instance. Alternatively, you can use the distcp command to copy the data to another HDFS cluster.935 936. Stop the HBase cluster937. Copy the HBase data directory to a backup location using the https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html[distcp command] as the HDFS super user (shown below on a security enabled cluster)938+939.Using distcp to backup the HBase data directory940[source,bash]941----942 943[hpnewton@gateway_node.example.com ~]$ kinit -k -t hdfs.keytab hdfs@EXAMPLE.COM944[hpnewton@gateway_node.example.com ~]$ hadoop distcp /hbase /hbase-pre-upgrade-backup945 946----947. Distcp will launch a mapreduce job to handle copying the files in a distributed fashion. Check the output of the distcp command to ensure this job completed successfully.948 949.Performing a rollback950 951. Stop HBase952. Perform a downgrade for HDFS and a downgrade/rollback for ZooKeeper (HBase should remain stopped)953. Change the installed version of HBase to the previous version954. Restore the HBase data directory from prior to the upgrade as the HDFS super user (shown below on a security enabled cluster). If you backed up your data on another HDFS cluster instead of locally, you will need to use the distcp command to copy it back to the current HDFS cluster.955+956.Restore the HBase data directory957[source,bash]958----959[hpnewton@gateway_node.example.com ~]$ kinit -k -t hdfs.keytab hdfs@EXAMPLE.COM960[hpnewton@gateway_node.example.com ~]$ hdfs dfs -mv /hbase /hbase-upgrade-rollback961[hpnewton@gateway_node.example.com ~]$ hdfs dfs -mv /hbase-pre-upgrade-backup /hbase962----963. Clean out ZooKeeper information related to HBase. WARNING: This step will permanently destroy all replication peers. Please see the section on HBase Replication under Caveats for more information.964+965.Clean HBase information out of ZooKeeper966[source,bash]967----968[hpnewton@gateway_node.example.com ~]$ zookeeper-client -server zookeeper1.example.com:2181,zookeeper2.example.com:2181,zookeeper3.example.com:2181969Welcome to ZooKeeper!970JLine support is disabled971rmr /hbase972quit973Quitting...974----975. Start HBase976. Verify HBase contents–use the HBase shell to list tables and scan some known values.977 978== Upgrade Paths979 980[[upgrade2.0]]981=== Upgrading from 1.x to 2.x982 983In this section we will first call out significant changes compared to the prior stable HBase release and then go over the upgrade process. Be sure to read the former with care so you avoid suprises.984 985==== Changes of Note!986 987First we'll cover deployment / operational changes that you might hit when upgrading to HBase 2.0+. After that we'll call out changes for downstream applications. Please note that Coprocessors are covered in the operational section. Also note that this section is not meant to convey information about new features that may be of interest to you. For a complete summary of changes, please see the CHANGES.txt file in the source release artifact for the version you are planning to upgrade to.988 989[[upgrade2.0.basic.requirements]]990.Update to basic prerequisite minimums in HBase 2.0+991As noted in the section <<basic.prerequisites>>, HBase 2.0+ requires a minimum of Java 8 and Hadoop 2.6. The HBase community recommends ensuring you have already completed any needed upgrades in prerequisites prior to upgrading your HBase version.992 993[[upgrade2.0.hbck]]994.HBCK must match HBase server version995You *must not* use an HBase 1.x version of HBCK against an HBase 2.0+ cluster. HBCK is strongly tied to the HBase server version. Using the HBCK tool from an earlier release against an HBase 2.0+ cluster will destructively alter said cluster in unrecoverable ways.996 997As of HBase 2.0, HBCK (A.K.A _HBCK1_ or _hbck1_) is a read-only tool that can report the status of some non-public system internals. You should not rely on the format nor content of these internals to remain consistent across HBase releases.998 999To read about HBCK's replacement, see <<HBCK2>> in <<ops_mgt>>.1000 1001 1002////1003Link to a ref guide section on HBCK in 2.0 that explains use and calls out the inability of clients and server sides to detect version of each other.1004////1005 1006[[upgrade2.0.removed.configs]]1007.Configuration settings no longer in HBase 2.0+1008 1009The following configuration settings are no longer applicable or available. For details, please see the detailed release notes.1010 1011* hbase.config.read.zookeeper.config (see <<upgrade2.0.zkconfig>> for migration details)1012* hbase.zookeeper.useMulti (HBase now always uses ZK's multi functionality)1013* hbase.rpc.client.threads.max1014* hbase.rpc.client.nativetransport1015* hbase.fs.tmp.dir1016// These next two seem worth a call out section?1017* hbase.bucketcache.combinedcache.enabled1018* hbase.bucketcache.ioengine no longer supports the 'heap' value.1019* hbase.bulkload.staging.dir1020* hbase.balancer.tablesOnMaster wasn't removed, strictly speaking, but its meaning has fundamentally changed and users should not set it. See the section <<upgrade2.0.regions.on.master>> for details.1021* hbase.master.distributed.log.replay See the section <<upgrade2.0.distributed.log.replay>> for details1022* hbase.regionserver.disallow.writes.when.recovering See the section <<upgrade2.0.distributed.log.replay>> for details1023* hbase.regionserver.wal.logreplay.batch.size See the section <<upgrade2.0.distributed.log.replay>> for details1024* hbase.master.catalog.timeout1025* hbase.regionserver.catalog.timeout1026* hbase.metrics.exposeOperationTimes1027* hbase.metrics.showTableName1028* hbase.online.schema.update.enable (HBase now always supports this)1029* hbase.thrift.htablepool.size.max1030 1031[[upgrade2.0.renamed.configs]]1032.Configuration properties that were renamed in HBase 2.0+1033 1034The following properties have been renamed. Attempts to set the old property will be ignored at run time.1035 1036.Renamed properties1037[options=""header""]1038|============================================================================================================1039|Old name |New name1040|hbase.rpc.server.nativetransport |hbase.netty.nativetransport1041|hbase.netty.rpc.server.worker.count |hbase.netty.worker.count1042|hbase.hfile.compactions.discharger.interval |hbase.hfile.compaction.discharger.interval1043|hbase.hregion.percolumnfamilyflush.size.lower.bound |hbase.hregion.percolumnfamilyflush.size.lower.bound.min1044|============================================================================================================1045 1046[[upgrade2.0.changed.defaults]]1047.Configuration settings with different defaults in HBase 2.0+1048 1049The following configuration settings changed their default value. Where applicable, the value to set to restore the behavior of HBase 1.2 is given.1050 1051* hbase.security.authorization now defaults to false. set to true to restore same behavior as previous default.1052* hbase.client.retries.number is now set to 10. Previously it was 35. Downstream users are advised to use client timeouts as described in section <<config_timeouts>> instead.1053* hbase.client.serverside.retries.multiplier is now set to 3. Previously it was 10. Downstream users are advised to use client timesout as describe in section <<config_timeouts>> instead.1054* hbase.master.fileSplitTimeout is now set to 10 minutes. Previously it was 30 seconds.1055* hbase.regionserver.logroll.multiplier is now set to 0.5. Previously it was 0.95. This change is tied with the following doubling of block size. Combined, these two configuration changes should make for WALs of about the same size as those in hbase-1.x but there should be less incidence of small blocks because we fail to roll the WAL before we hit the blocksize threshold. See link:https://issues.apache.org/jira/browse/HBASE-19148[HBASE-19148] for discussion.1056* hbase.regionserver.hlog.blocksize defaults to 2x the HDFS default block size for the WAL dir. Previously it was equal to the HDFS default block size for the WAL dir.1057* hbase.client.start.log.errors.counter changed to 5. Previously it was 9.1058* hbase.ipc.server.callqueue.type changed to 'fifo'. In HBase versions 1.0 - 1.2 it was 'deadline'. In prior and later 1.x versions it already defaults to 'fifo'.1059* hbase.hregion.memstore.chunkpool.maxsize is 1.0 by default. Previously it was 0.0. Effectively, this means previously we would not use a chunk pool when our memstore is onheap and now we will. See the section <<gcpause>> for more infromation about the MSLAB chunk pool.1060* hbase.master.cleaner.interval is now set to 10 minutes. Previously it was 1 minute.1061* hbase.master.procedure.threads will now default to 1/4 of the number of available CPUs, but not less than 16 threads. Previously it would be number of threads equal to number of CPUs.1062* hbase.hstore.blockingStoreFiles is now 16. Previously it was 10.1063* hbase.http.max.threads is now 16. Previously it was 10.1064* hbase.client.max.perserver.tasks is now 2. Previously it was 5.1065* hbase.normalizer.period is now 5 minutes. Previously it was 30 minutes.1066* hbase.regionserver.region.split.policy is now SteppingSplitPolicy. Previously it was IncreasingToUpperBoundRegionSplitPolicy.1067* replication.source.ratio is now 0.5. Previously it was 0.1.1068 1069[[upgrade2.0.regions.on.master]]1070.""Master hosting regions"" feature broken and unsupported1071 1072The feature ""Master acts as region server"" and associated follow-on work available in HBase 1.y is non-functional in HBase 2.y and should not be used in a production setting due to deadlock on Master initialization. Downstream users are advised to treat related configuration settings as experimental and the feature as inappropriate for production settings.1073 1074A brief summary of related changes:1075 1076* Master no longer carries regions by default1077* hbase.balancer.tablesOnMaster is a boolean, default false (if it holds an HBase 1.x list of tables, will default to false)1078* hbase.balancer.tablesOnMaster.systemTablesOnly is boolean to keep user tables off master. default false1079* those wishing to replicate old list-of-servers config should deploy a stand-alone RegionServer process and then rely on Region Server Groups1080 1081[[upgrade2.0.distributed.log.replay]]1082.""Distributed Log Replay"" feature broken and removed1083 1084The Distributed Log Replay feature was broken and has been removed from HBase 2.y+. As a consequence all related configs, metrics, RPC fields, and logging have also been removed. Note that this feature was found to be unreliable in the run up to HBase 1.0, defaulted to being unused, and was effectively removed in HBase 1.2.0 when we started ignoring the config that turns it on (link:https://issues.apache.org/jira/browse/HBASE-14465[HBASE-14465]). If you are currently using the feature, be sure to perform a clean shutdown, ensure all DLR work is complete, and disable the feature prior to upgrading.1085 1086[[upgrade2.0.prefix-tree.removed]]1087._prefix-tree_ encoding removed1088 1089The prefix-tree encoding was removed from HBase 2.0.0 (link:https://issues.apache.org/jira/browse/HBASE-19179[HBASE-19179]).1090It was (late!) deprecated in hbase-1.2.7, hbase-1.4.0, and hbase-1.3.2.1091 1092This feature was removed because it as not being actively maintained. If interested in reviving this1093sweet facility which improved random read latencies at the expensive of slowed writes,1094write the HBase developers list at _dev at hbase dot apache dot org_.1095 1096The prefix-tree encoding needs to be removed from all tables before upgrading to HBase 2.0+.1097To do that first you need to change the encoding from PREFIX_TREE to something else that is supported in HBase 2.0.1098After that you have to major compact the tables that were using PREFIX_TREE encoding before.1099To check which column families are using incompatible data block encoding you can use <<ops.pre-upgrade,Pre-Upgrade Validator>>.1100 1101[[upgrade2.0.metrics]]1102.Changed metrics1103 1104The following metrics have changed names:1105 1106* Metrics previously published under the name ""AssignmentManger"" [sic] are now published under the name ""AssignmentManager""1107 1108The following metrics have changed their meaning:1109 1110* The metric 'blockCacheEvictionCount' published on a per-region server basis no longer includes blocks removed from the cache due to the invalidation of the hfiles they are from (e.g. via compaction).1111* The metric 'totalRequestCount' increments once per request; previously it incremented by the number of `Actions` carried in the request; e.g. if a request was a `multi` made of four Gets and two Puts, we'd increment 'totalRequestCount' by six; now we increment by one regardless. Expect to see lower values for this metric in hbase-2.0.0.1112* The 'readRequestCount' now counts reads that return a non-empty row where in older hbases, we'd increment 'readRequestCount' whether a Result or not. This change will flatten the profile of the read-requests graphs if requests for non-existent rows. A YCSB read-heavy workload can do this dependent on how the database was loaded.1113 1114The following metrics have been removed:1115 1116* Metrics related to the Distributed Log Replay feature are no longer present. They were previsouly found in the region server context under the name 'replay'. See the section <<upgrade2.0.distributed.log.replay>> for details.1117 1118The following metrics have been added:1119 1120* 'totalRowActionRequestCount' is a count of region row actions summing reads and writes.1121 1122[[upgrade2.0.logging]]1123.Changed logging1124HBase-2.0.0 now uses link:https://www.slf4j.org/[slf4j] as its logging frontend.1125Prevously, we used link:http://logging.apache.org/log4j/1.2/[log4j (1.2)].1126For most the transition should be seamless; slf4j does a good job interpreting1127_log4j.properties_ logging configuration files such that you should not notice1128any difference in your log system emissions.1129 1130That said, your _log4j.properties_ may need freshening. See link:https://issues.apache.org/jira/browse/HBASE-20351[HBASE-20351]1131for example, where a stale log configuration file manifest as netty configuration1132being dumped at DEBUG level as preamble on every shell command invocation.1133 1134[[upgrade2.0.zkconfig]]1135.ZooKeeper configs no longer read from zoo.cfg1136 1137HBase no longer optionally reads the 'zoo.cfg' file for ZooKeeper related configuration settings. If you previously relied on the 'hbase.config.read.zookeeper.config' config for this functionality, you should migrate any needed settings to the hbase-site.xml file while adding the prefix 'hbase.zookeeper.property.' to each property name.1138 1139[[upgrade2.0.permissions]]1140.Changes in permissions1141The following permission related changes either altered semantics or defaults:1142 1143* Permissions granted to a user now merge with existing permissions for that user, rather than over-writing them. (see link:https://issues.apache.org/jira/browse/HBASE-17472[the release note on HBASE-17472] for details)1144* Region Server Group commands (added in 1.4.0) now require admin privileges.1145 1146[[upgrade2.0.admin.commands]]1147.Most Admin APIs don't work against an HBase 2.0+ cluster from pre-HBase 2.0 clients1148 1149A number of admin commands are known to not work when used from a pre-HBase 2.0 client. This includes an HBase Shell that has the library jars from pre-HBase 2.0. You will need to plan for an outage of use of admin APIs and commands until you can also update to the needed client version.1150 1151The following client operations do not work against HBase 2.0+ cluster when executed from a pre-HBase 2.0 client:1152 1153* list_procedures1154* split1155* merge_region1156* list_quotas1157* enable_table_replication1158* disable_table_replication1159* Snapshot related commands1160 1161.Deprecated in 1.0 admin commands have been removed.1162 1163The following commands that were deprecated in 1.0 have been removed. Where applicable the replacement command is listed.1164 1165* The 'hlog' command has been removed. Downstream users should rely on the 'wal' command instead.1166 1167[[upgrade2.0.memory]]1168.Region Server memory consumption changes.1169 1170Users upgrading from versions prior to HBase 1.4 should read the instructions in section <<upgrade1.4.memory>>.1171 1172Additionally, HBase 2.0 has changed how memstore memory is tracked for flushing decisions. Previously, both the data size and overhead for storage were used to calculate utilization against the flush threashold. Now, only data size is used to make these per-region decisions. Globally the addition of the storage overhead is used to make decisions about forced flushes.1173 1174[[upgrade2.0.ui.splitmerge.by.row]]1175.Web UI for splitting and merging operate on row prefixes1176 1177Previously, the Web UI included functionality on table status pages to merge or split based on an encoded region name. In HBase 2.0, instead this functionality works by taking a row prefix.1178 1179[[upgrade2.0.replication]]1180.Special upgrading for Replication users from pre-HBase 1.41181 1182User running versions of HBase prior to the 1.4.0 release that make use of replication should be sure to read the instructions in the section <<upgrade1.4.replication>>.1183 1184[[upgrade2.0.shell]]1185.HBase shell changes1186 1187The HBase shell command relies on a bundled JRuby instance. This bundled JRuby been updated from version 1.6.8 to version 9.1.10.0. The represents a change from Ruby 1.8 to Ruby 2.3.3, which introduces non-compatible language changes for user scripts.1188 1189The HBase shell command now ignores the '--return-values' flag that was present in early HBase 1.4 releases. Instead the shell always behaves as though that flag were passed. If you wish to avoid having expression results printed in the console you should alter your IRB configuration as noted in the section <<irbrc>>.1190 1191[[upgrade2.0.coprocessors]]1192.Coprocessor APIs have changed in HBase 2.0+1193 1194All Coprocessor APIs have been refactored to improve supportability around binary API compatibility for future versions of HBase. If you or applications you rely on have custom HBase coprocessors, you should read link:https://issues.apache.org/jira/browse/HBASE-18169[the release notes for HBASE-18169] for details of changes you will need to make prior to upgrading to HBase 2.0+.1195 1196For example, if you had a BaseRegionObserver in HBase 1.2 then at a minimum you will need to update it to implement both RegionObserver and RegionCoprocessor and add the method1197 1198[source,java]1199----1200...

Showing the first 1,200 of 79791 lines. Download the file for the rest.