DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.

Apache Kylin

Space shortcuts

Apache Kylin : Analytical Data Warehouse for Big Data

Page tree

Welcome to Kylin Wiki.


Property
Importance
DatatypeConfiguration LevelDefaultDescriptionVersionReference
kylin.engine.spark.build-class-name

TRIVIAL

String

PROCESS

org.apache.kylin.engine.spark.job.CubeBuildJob
For developer only. The className use in spark-submit

4.0.0-ALPHA


kylin.engine.spark.cluster-info-fetcher-class-name

TRIVIAL

String

PROCESS

org.apache.kylin.cluster.YarnInfoFetcher
For developer only. Fetch yarn information of spark job

4.0.0-ALPHA


kylin.engine.spark-conf.XXX

MINOR

String

PROCESS


  1. Before Kylin submit a cubing job, some major property(cores and memory) will be automatically adjusted adaptively. (if kylin.spark-conf.auto.prior was set to true).
  2. After auto adjust, spark conf will be overwrite by this property. If you want to set spark.driver.extraJavaOptions=-Dhdp.version=current, you can add follow line in kylin.properties:
kylin.engine.spark-conf.spark.driver.extraJavaOptions=-Dhdp.version=current

4.0.0-ALPHA

Adaptively-adjust-spark-parameters
kylin.storage.provider

TRIVIAL

String

PROCESS

org.apache.kylin.common.storage.DefaultStorageProvider

The content summary objects returned by different cloud vendors are not the same, so need to provide targeted implementation.

You can refer to this to learn more : org.apache.kylin.common.storage.IStorageProvider

4.0.0-ALPHA


kylin.engine.spark.merge-class-name

TRIVIAL

String

PROCESS

org.apache.kylin.engine.spark.job.CubeMergeJob
For developer only. The className use in spark-submit

4.0.0-ALPHA


kylin.engine.spark.task-impact-instance-enabled

TRIVIAL

Boolean

PROCESS

true

UPDATING

4.0.0-ALPHA

Adaptively-adjust-spark-parameters

kylin.engine.spark.task-core-factor

MEDIUM

Integer

PROCESS

3

UPDATING

4.0.0-ALPHA


kylin.engine.driver-memory-base

MEDIUM

Integer

PROCESS

1024Auto adujst spark.driver.memory for Build Engine if kylin.engine.spark-conf.spark.driver.memory is not set.



4.0.0-ALPHA

Adaptively-adjust-spark-parameters
kylin.engine.driver-memory-strategy

MEDIUM

Array

PROCESS

{"2", "20", "100"}
UPDATING

4.0.0-ALPHA


kylin.engine.driver-memory-maximum

MEDIUM

Integer

PROCESS

4096

UPDATING

4.0.0-ALPHA


kylin.engine.persist-flattable-threshold

MEDIUM

Integer

PROCESS

1If the number of cuboids which will be build from flat table is bigger than this threshold, the flat table will be persisted into $HDFS_WORKING_DIR/job_tmp/flat_table for saving more memory.

4.0.0-ALPHA


kylin.snapshot.parallel-build-timeout-seconds

MAJOR

Integer

PROCESS

3600
To improve the speed of snapshot build.


4.0.0-ALPHA


kylin.snapshot.parallel-build-enabled

MAJOR

Boolean

PROCESS

true

UPDATING











kylin.spark-conf.auto.prior

MINOR

Boolean

PROCESS

true Enable adjust spark parameters adaptively.

4.0.0-ALPHA

Adaptively-adjust-spark-parameters
kylin.engine.submit-hadoop-conf-dir

MAJOR

String

PROCESS

/etc/hadoop/conf

Set HADOOP_CONF_DIR for spark-submit.

4.0.0-ALPHA


kylin.storage.columnar.shard-size-mb

MAJOR

Integer

CUBE

128

The max size of pre-calcualted cuboid parquet file.

4.0.0-ALPHA

ShardBy
kylin.storage.columnar.shard-rowcount

MAJOR

Long

CUBE

2500000

The max rows of pre-calcualted cuboid parquet file.

4.0.0-ALPHA


kylin.storage.columnar.shard-countdistinct-rowcount

MAJOR

Long

CUBE

1000000The max rows of pre-calcualted cuboid parquet file when cuboid has bitmap measure. (When cuboid has BItmap, it is large.)

4.0.0-ALPHA


kylin.query.spark-engine.join-memory-fraction

MEDIUM

Double

PROCESS

0.3Limit memory used by broadcast join of Sparder. (Broadcast join cause unstable.)

4.0.0-ALPHA


File NameContentComment
cubing_detect_items.json

sampling_detect_items.json



count_distinct.json

true/false

resource_paths.json






  • No labels

Bring OLAP Back to Big Data!