Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This page mainly provides some new configuration items of kylin 4.0, rather than complete configuration documents.

...

kylin.env

...

kylin.env.hdfs-working-dir

...

/kylin

...

Working folder in HDFS, better be qualified absolute path, make sure user has the right permission to this directory

...

kylin.env.zookeeper-base-path

...

kylin zk base path

...

kylin.env.zookeeper-is-local

...

Run a TestingServer for curator locally

...

kylin.env.zookeeper-connect-string

...

sandbox.hortonworks.com

...

Connect to a remote zookeeper with the url, should set kylin.env.zookeeper-is-local to false

...

kylin.env.hadoop-conf-dir

...

/etc/hadoop/conf

...

Hadoop conf folder, will export this as "HADOOP_CONF_DIR" to run spark-submit. This must contain site xmls of core, yarn, hive, and hbase in one folder

...

kylin.env.hdfs-metastore-bigcell-dir

...

kylin.env.hdfs-metastore-bigcell-dir

...

kylin.env.zookeeper-base-sleep-time

...

kylin.env.zookeeper.zk-auth

...

digest:ADMIN:KYLIN

...

kylin.env.zookeeper-acl-enabled

...

kylin.job.zookeeper-monitor-interval

...

kylin.job.log-dir

...

kylin.job.use-remote-cli

...

kylin.job.remote-cli-port

...

kylin.job.remote-cli-hostname

...

kylin.job.remote-cli-username

...

kylin.job.remote-cli-password

...

kylin.job.remote-cli-working-dir

...

kylin.job.allow-empty-segment

...

kylin.job.max-concurrent-jobs

...

kylin.job.sampling-percentage

...

kylin.job.dependency-filter-list

...

"[^,]*hive-exec[^,]*?\\.jar" + "|"
+ "[^,]*hive-metastore[^,]*?\\.jar" + "|" + "[^,]*hive-hcatalog-core[^,]*?\\.jar"

...

kylin.job.notification-enabled

...

kylin.job.notification-mail-enable-starttls

...

kylin.job.notification-mail-port

...

kylin.job.notification-mail-host

...

kylin.job.notification-mail-username

...

kylin.job.notification-mail-password

...

kylin.job.notification-mail-sender

...

kylin.job.notification-admin-emails

...

kylin.job.retry

...

kylin.job.retry-interval

...

kylin.job.retry-exception-classes

...

kylin.job.sampling-hll-precision

...

kylin.job.scheduler.default

...

kylin.job.scheduler.priority-considered

...

kylin.job.scheduler.priority-bar-fetch-from-queue

...

kylin.job.scheduler.poll-interval-second

...

kylin.job.scheduler.safemode

...

kylin.job.scheduler.safemode.runnable-projects

...

kylin.job.error-record-threshold

...

kylin.job.use-advanced-flat-table

...

kylin.job.advanced-flat-table.class

...

kylin.job.tracking-url-pattern

...

kylin.job.metadata-persist-retry

...

kylin.job.cube-auto-ready-enabled

...

kylin.job.cube-inmem-builder-class

...

org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder

...

kylin.job.execute-output.max-size

...

10485760

...

kylin.metadata.url

...

kylin_metadata@hbase

...

The metadata store in hbase

...

kylin.metadata.sync-retries

...

metadata cache sync retry times

...

kylin.metadata.sync-error-handler

...

MockupErrHandler

...

kylin.server.mode

...

Kylin server mode, valid value [all, query, job]

...

kylin.server.cluster-servers

...

List of web servers in use, this enables one web server instance to sync up with other servers.

...

kylin.server.query-metrics-percentiles-intervals

...

60, 360, 3600

...

kylin query metrics percentiles intervals default=60, 300, 3600

...

kylin.server.host-address

...

kylin.server.cluster-name

...

kylin.server.init-tasks

...

kylin.server.sequence-sql.workers-per-server

...

kylin.server.sequence-sql.expire-time

...

kylin.server.query-metrics-enabled

...

kylin.server.query-metrics2-enabled

...

kylin.server.auth-user-cache.expire-seconds

...

kylin.server.auth-user-cache.max-entries

...

kylin.server.external-acl-provider

...

kylin.server.cluster-servers-with-mode

...

kylin.storage.hbase.owner-tag

...

whoami@kylin.apache.org

...

Optional information for the owner of kylin platform, it can be your team's email, Currently it will be attached to each kylin's htable attribute

...

kylin.storage.url

...

The storage for final cube file in hbase

...

kylin.storage.hbase.cluster-fs

...

HBase Cluster FileSystem, which serving hbase, format as hdfs://hbase-cluster:8020, Leave empty if hbase running on same cluster with hive and mapreduce

...

kylin.storage.hbase.compression-codec

...

default compression codec for htable,snappy,lzo,gzip,lz4

...

kylin.storage.hbase.cluster-hdfs-config-file

...

kylin.storage.hbase.region-cut-gb

...

The cut size for hbase region, in GB. E.g, for cube whose capacity be marked as "SMALL", split region per 10GB by default

...

kylin.storage.hbase.min-region-count

...

kylin.storage.hbase.max-region-count

...

kylin.storage.hbase.hfile-size-gb

...

The hfile size of GB, smaller hfile leading to the converting hfile MR has more reducers and be faster. set 0 to disable this optimization

...

kylin.storage.hbase.run-local-coprocessor

...

kylin.storage.hbase.coprocessor-mem-gb

...

kylin.storage.partition.aggr-spill-enabled

...

kylin.storage.partition.max-scan-bytes

...

3L * 1024 * 1024 * 1024

...

kylin.storage.hbase.coprocessor-timeout-seconds

...

kylin.storage.hbase.max-fuzzykey-scan

...

kylin.storage.hbase.max-fuzzykey-scan-split

...

kylin.storage.hbase.max-visit-scanrange

...

1000000

...

kylin.storage.hbase.gtstorage

...

org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC

...

kylin.storage.hbase.scan-cache-rows

...

kylin.storage.hbase.region-cut-gb

...

kylin.storage.hbase.max-scan-result-bytes

...

5 * 1024 * 1024

...

kylin.storage.hbase.compression-codec

...

kylin.storage.hbase.rowkey-encoding

...

kylin.storage.hbase.block-size-bytes

...

kylin.storage.hbase.small-family-block-size-bytes

...

kylin.storage.hbase.endpoint-compress-result

...

kylin.storage.hbase.max-hconnection-threads

...

kylin.storage.hbase.core-hconnection-threads

...

kylin.storage.hbase.hconnection-threads-alive-seconds

...

kylin.storage.hbase.replication-scope

...

kylin.storage.clean-after-delete-operation

...

kylin.storage.project-isolation-enable

...

kylin.storage.limit-push-down-enabled

...

kylin.storage.default

...

kylin.storage.hbase.table-name-prefix

...

kylin.web.timezone

...

Display timezone on UI,format like[GMT+N or GMT-N]

...

kylin.web.help.length

...

kylin.web.help.0

...

More configuration items can be viewed: 

http://kylin.apache.org/docs/

...

install/

...

configuration.html

...

.

...

odbc|ODBC Driver|http://kylin.apache.org/docs/tutorial/odbc.html

...

kylin.web.help.2

...

tableau|Tableau Guide|http://kylin.apache.org/docs/tutorial/tableau_91.html

...

kylin.web.help.3

...

onboard|Cube Design Tutorial|http://kylin.apache.org/docs/howto/howto_optimize_cubes.html

...

kylin.web.export-allow-admin

...

kylin.web.export-allow-other

...

kylin.web.hide-measures

...

kylin.web.default-time-filter

...

kylin.web.cross-domain-enabled

...

kylin.web.dashboard-enabled

...

kylin.engine.mr.config-override.

...

kylin.engine.mr.mem-hungry-config-override.

...

kylin.engine.mr.uhc-config-override.

...

kylin.engine.mr.base-cuboid-config-override.

...

kylin.engine.mr.reduce-input-mb

...

kylin.engine.mr.reduce-count-ratio

...

kylin.engine.mr.min-reducer-number

...

kylin.engine.mr.max-reducer-number

...

kylin.engine.mr.mapper-input-rows

...

1000000

...

kylin.engine.mr.max-cuboid-stats-calculator-number

...

set 1 to disable multi-thread statistics calculation

...

kylin.engine.mr.cuboid-number-per-stats-calculator

...

kylin.engine.mr.per-reducer-hll-cuboid-number

...

kylin.engine.mr.hll-max-reducer-number

...

kylin.engine.mr.build-dict-in-reducer

...

kylin.engine.mr.yarn-check-status-url

...

kylin.engine.mr.yarn-check-interval-seconds

...

kylin.engine.mr.use-local-classpath

...

kylin.engine.mr.uhc-config-override.mapreduce.reduce.memory.mb

...

kylin.engine.mr.uhc-config-override.mapred.reduce.child.java.opts

...

kylin.engine.mr.yarn-check-interval-seconds

...

Time interval to check hadoop job status

...

kylin.engine.mr.max-reducer-number

...

kylin.engine.mr.uhc-reducer-count

...

kylin.engine.mr.lib-dir

...

kylin.engine.spark-conf.

...

getSparkConfigOverride

...

kylin.engine.spark.additional-jars

...

kylin.engine.spark.rdd-partition-cut-mb

...

kylin.engine.spark.min-partition

...

kylin.engine.spark.max-partition

...

kylin.engine.spark.storage-level

...

kylin.engine.spark.sanity-check-enabled

...

kylin.engine.spark-fact-distinct

...

kylin.engine.spark-uhc-dictionary

...

kylin.engine.spark-cardinality

...

kylin.engine.spark.output.max-size

...

kylin.engine.spark-dimension-dictionary

...

kylin.engine.spark-create-table-enabled

...

kylin.engine.flink-conf.

...

kylin.engine.flink.additional-jars

...

kylin.engine.flink.partition-cut-mb

...

kylin.engine.flink.min-partition

...

kylin.engine.flink.max-partition

...

kylin.engine.flink.sanity-check-enabled

...

kylin.engine.livy-conf.livy-enabled

...

kylin.engine.livy.backtick.quote

...

kylin.engine.livy-conf.livy-url

...

kylin.engine.livy-conf.livy-key.

...

kylin.engine.livy-conf.livy-arr.

...

kylin.engine.livy-conf.livy-map.

...

kylin.engine.default

...

kylin.query.enable-dict-enumerator

...

kylin.query.calcite.enumerable-rules-enabled

...

kylin.query.calcite.reduce-rules-enabled

...

kylin.query.convert-create-table-to-with

...

kylin.query.calcite.extras-props.

...

kylin.query.calcite.add-rule

...

kylin.query.calcite.remove-rule

...

kylin.query.enable-dynamic-column

...

kylin.query.skip-empty-segments

...

kylin.query.disable-cube-noagg-sql

...

kylin.query.stream-aggregate-enabled

...

kylin.storage.project-isolation-enable

...

kylin.query.max-limit-pushdown

...

kylin.query.force-limit

...

kylin.query.scan-threshold

...

10000000

...

kylin.query.lazy-query-enabled

...

kylin.query.lazy-query-waiting-timeout-milliseconds

...

kylin.query.project-concurrent-running-threshold

...

kylin.query.max-scan-bytes

...

kylin.query.max-return-rows

...

5000000

...

kylin.query.translated-in-clause-max-size

...

kylin.query.derived-filter-translation-threshold

...

kylin.query.badquery-stacktrace-depth

...

kylin.query.badquery-history-number

...

kylin.query.badquery-alerting-seconds

...

kylin.query.timeout-seconds-coefficient

...

kylin.query.badquery-persistent-enabled

...

kylin.query.transformers

...

kylin.query.transformers

...

kylin.query.cache-threshold-duration

...

kylin.query.cache-threshold-scan-count

...

kylin.query.cache-threshold-scan-bytes

...

kylin.query.security-enabled

...

kylin.query.cache-enabled

...

kylin.query.ignore-unknown-function

...

kylin.cache.memcached.hosts

...

kylin.query.segment-cache-enabled

...

kylin.query.segment-cache-timeout

...

kylin.query.segment-cache-max-size

...

kylin.query.access-controller

...

kylin.query.statement-cache-max-num

...

kylin.query.statement-cache-max-num-per-key

...

kylin.query.statement-cache-enabled

...

kylin.query.max-dimension-count-distinct

...

5000000

...

kylin.query.timeout-seconds

...

kylin.query.pushdown.enabled

...

kylin.query.pushdown.update-enabled

...

kylin.query.schema-factory

...

org.apache.kylin.query.schema.OLAPSchemaFactory

...

kylin.query.pushdown.runner-class-name

...

kylin.query.pushdown.runner.ids

...

kylin.query.pushdown.converter-class-names

...

kylin.query.pushdown.cache-enabled

...

kylin.query.pushdown.jdbc.url

...

kylin.query.pushdown.jdbc.driver

...

kylin.query.pushdown.jdbc.username

...

kylin.query.pushdown.jdbc.password

...

kylin.query.pushdown.jdbc.pool-max-total

...

kylin.query.pushdown.jdbc.pool-max-idle

...

kylin.query.pushdown.jdbc.pool-min-idle

...

kylin.query.security.table-acl-enabled

...

kylin.query.escape-default-keyword

...

kylin.query.realization-filter

...

kylin.query.signature-class

...

org.apache.kylin.rest.signature.FactTableRealizationSetCalculator

...

kylin.query.cache-signature-enabled

...

kylin.query.flat-filter-max-children

...

Configurations new added 

...

kylin.engine.spark.build-class-name

...

org.apache.kylin.engine.spark.job.CubeBuildJob

...

kylin.engine.spark.task-impact-instance-enabled

...

kylin.engine.spark.task-core-factor

...

kylin.engine.driver-memory-base

...

kylin.engine.driver-memory-strategy

...

 {"2", "20", "100" }

...

Auto adjust the memory of driver


PartLinkDescription
Build EngineKylin 4.0 Build Engine ConfigurationConfiguration entry for build engine related behavior in kylin.properties.
Query EngineKylin 4.0 Query Engine ConfigurationConfiguration entry for query engine related behavior in kylin.properties.
SecurityTo be updated

Some configuration not appeared in kylin.properties, included :

  1. applicationContext.xml
  2. kylin-metrics.xml
  3. kylin.security.xml
  4. setenv.sh 
  5. cache.xml
  6. etc.
Logging ConfigurationLogger configuration

...

kylin.engine.driver-memory-maximum

...

kylin.engine.persist-flattable-threshold

...

kylin.engine.spark.cluster-info-fetcher-class-name

...

org.apache.kylin.cluster.YarnInfoFetcher

...

kylin.engine.spark.merge-class-name

...

org.apache.kylin.engine.spark.job.CubeMergeJob

...

kylin.engine.max-retry-time

...

kylin.engine.retry-memory-gradient

...

kylin.engine.retry-overheadMemory-gradient

...

kylin.engine.max-allocation-proportion

...

kylin.engine.base-executor-instance

...

kylin.engine.executor-instance-strategy

...

100,2,500,3,1000,4

...

kylin.engine.submit-hadoop-conf-dir

...

kylin.snapshot.parallel-build-enabled

...

kylin.snapshot.parallel-build-timeout-seconds

...

kylin.snapshot.shard-size-mb

...

kylin.storage.provider

...

org.apache.kylin.common.storage.DefaultStorageProvider

...

kylin.storage.columnar.shard-size-mb

...

kylin.storage.columnar.shard-rowcount

...

2500000

...

kylin.storage.columnar.shard-countdistinct-rowcount

...

1000000

...

kylin.storage.columnar.repartition-threshold-size-mb

...

kylin.storage.columnar.shard-min

...

kylin.storage.columnar.shard-max

...

kylin.storage.columnar.hdfs-blocksize-bytes

...

5 * shard_size

...

kylin.storage.columnar.shard-expand-factor

...

kylin.storage.columnar.dfs-replication

...

kylin.spark-conf.auto.prior

...

kylin.job.log-print-enabled

...

kylin.query.spark-engine.join-memory-fraction

...

driver memory that can be used by join(mostly BHJ)

...

kylin.query.spark-engine.enabled

...

true

...

kylin.query.spark-engine.partition-split-size-mb

...

kylin.query.spark-engine.expose-sharding-trait

...

kylin.query.spark-engine.spark-sql-shuffle-partitions

...

kylin.query.spark-conf.

...

kylin.query.engine.sparder-additional-files

...

kylin.query.engine.sparder-additional-jars

...

kylin.query.pushdown.auto-set-shuffle-partitions-enabled

...

kylin.query.pushdown.base-shuffle-partition-size

...

kylin.query.intersect.separator

...

kylin.kerberos.enabled

...

kylin.kerberos.keytab

...

kylin.kerberos.zookeeper.server.principal

...

kylin.kerberos.ticket.refresh.interval.minutes

...

kylin.kerberos.monitor.interval.minutes

...

kylin.kerberos.platform

...

kylin.platform.zk.kerberos.enable

...

kylin.kerberos.krb5.conf

...

krb5.conf

...

kylin.kerberos.jaas.conf

...

jaas.conf

...

kylin.kerberos.principal

Configurations remained from Kylin3.x

...

Configurations removed from Kylin3.X

ModulePropertyDescription