DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.

Apache Kylin

Space shortcuts

Apache Kylin : Analytical Data Warehouse for Big Data

Page tree

Welcome to Kylin Wiki.


Property
Importance
DefaultDescriptionVersion
kylin.engine.spark.build-class-name

TRIVIAL

org.apache.kylin.engine.spark.job.CubeBuildJob
For developer only. The className use in spark-submit

4.0.0-ALPHA

kylin.engine.spark.cluster-info-fetcher-class-name

TRIVIAL

org.apache.kylin.cluster.YarnInfoFetcher
For developer only. Fetch yarn information of spark job

4.0.0-ALPHA

kylin.engine.spark-conf.XXX

MINOR


  1. Before Kylin submit a cubing job, some major property(cores and memory) will be automatically adjusted adaptively. (if kylin.spark-conf.auto.prior was set to true).
  2. After auto adjust, spark conf will be overwrite by this property. If you want to set spark.driver.extraJavaOptions=-Dhdp.version=current, you can add follow line in kylin.properties:
kylin.engine.spark-conf.spark.driver.extraJavaOptions=-Dhdp.version=current

4.0.0-ALPHA

kylin.storage.provider

TRIVIAL

org.apache.kylin.common.storage.DefaultStorageProvider

The content summary objects returned by different cloud vendors are not the same, so need to provide targeted implementation.

You can refer to this to learn more : org.apache.kylin.common.storage.IStorageProvider

4.0.0-ALPHA

kylin.engine.spark.merge-class-name

TRIVIAL

org.apache.kylin.engine.spark.job.CubeMergeJob
For developer only. The className use in spark-submit

4.0.0-ALPHA

kylin.engine.spark.task-impact-instance-enabled

TRIVIAL

true

UPDATING

4.0.0-ALPHA

kylin.engine.spark.task-core-factor

MEDIUM

3

UPDATING

4.0.0-ALPHA

kylin.engine.driver-memory-base

MEDIUM

1024Auto adujst spark.driver.memory for Build Engine if kylin.engine.spark-conf.spark.driver.memory is not set.



4.0.0-ALPHA

kylin.engine.driver-memory-strategy

MEDIUM

{"2", "20", "100"}
UPDATING

4.0.0-ALPHA

kylin.engine.driver-memory-maximum

MEDIUM

4096

UPDATING

4.0.0-ALPHA

kylin.engine.persist-flattable-threshold

MEDIUM

1If the number of cuboids which will be build from flat table is bigger than this threshold, the flat table will be persisted into $HDFS_WORKING_DIR/job_tmp/flat_table for saving more memory.

4.0.0-ALPHA

kylin.snapshot.parallel-build-timeout-seconds

MAJOR

3600
To improve the speed of snapshot build.


4.0.0-ALPHA

kylin.snapshot.parallel-build-enabled

MAJOR

true

UPDATING







kylin.spark-conf.auto.prior

MINOR

true Enable adjust spark parameters adaptively.

4.0.0-ALPHA

kylin.engine.submit-hadoop-conf-dir

MAJOR

/etc/hadoop/conf

Set HADOOP_CONF_DIR for spark-submit.

4.0.0-ALPHA

kylin.storage.columnar.shard-size-mb

MAJOR

128

The max size of pre-calcualted cuboid parquet file.

4.0.0-ALPHA

kylin.storage.columnar.shard-rowcount

MAJOR

2500000

The max rows of pre-calcualted cuboid parquet file.

4.0.0-ALPHA

kylin.storage.columnar.shard-countdistinct-rowcount

MAJOR

1000000The max rows of pre-calcualted cuboid parquet file when cuboid has bitmap measure. (When cuboid has BItmap, it is large.)

4.0.0-ALPHA

kylin.query.spark-engine.join-memory-fraction

MEDIUM

0.3Limit memory used by broadcast join of Sparder. (Broadcast join cause unstable.)

4.0.0-ALPHA

  • No labels

Bring OLAP Back to Big Data!