DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.

Apache Kylin : Analytical Data Warehouse for Big Data

Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Advanced Tables - Table Plus
sortDescendingtrue
sortColumnPriority
autoNumberSorttrue
allowExporttrue
displayDataFiltertrue
sortIcontrue


NONONOtrueNOYESYESYESNONONONOtitleYEStitleNO
PropertyRequiredPriority
Importance
DatatypeConfiguration LevelDefaultDescriptionVersionReference
kylin.engine.spark.build-class-name
Status
subtletrue
title

Status
subtletrue
titleMinor

String

Status
subtletrue
titleProcess

org.apache.kylin.engine.spark.job.CubeBuildJob
For developer only. The className use in spark-submit.

Status
subtletrue
colourBlue
title4.0.0-alpha


kylin.engine.spark.cluster-info-fetcher-class-name
Status
subtletrue
title

Status
subtletrue
titleMinor

String

Status
subtletrue
titleProcess

org.apache.kylin.cluster.YarnInfoFetcher
For developer only. Fetch yarn information of spark job

Status
subtletrue
colourBlue
title4.0.0-alpha


kylin.engine.spark-conf.XXX
Status
subtletrue
title

Status
subtletrue
titleMinor

String

Status
subtletrue
titleProcess

NullSpark configurations want to override for build job like "spark.driver.cores". If don't set these spark properties, kylin will automaticly adjust these properties before submitting build job. 

Status
subtletrue
colourBlue
title4.0.0-alpha

Adaptively-adjust-spark-parameters
kylin.storage.provider

Status
subtletrue
title

NO

status
subtletrue
titleMinor

String

Status
subtletrue
titleProcess

org.apache.kylin.common.storage.DefaultStorageProvider

The content summary objects returned by different cloud vendors are not the same, so need to provide targeted implementation.

You can refer to this to learn more : org.apache.kylin.common.storage.IStorageProvider

Status
subtletrue
colourBlue
title4.0.0-alpha


kylin.engine.spark.merge-class-name

Status
subtletrue
title

NO

Status
subtletrue
titleMinor

String

Status
subtletrue
titleProcess

org.apache.kylin.engine.spark.job.CubeMergeJob
For developer only. The className use in spark-submit

Status
subtletrue
colourBlue
title4.0.0-alpha


kylin.engine.spark.task-impact-instance-enabled
Status
subtle
titleNO

Boolean

Status
subtletrue
titleProcess

trueCheck kylin.engine.spark.task-core-factor. If kylin.engine.spark.task-impact-instance-enabled is set to true and kylin.engine.spark-conf.spark.executor.instances is not set, Kylin will calculate spark.executor.instances for Build Engine.

Status
subtletrue
colourBlue
title4.0.0-alpha

Adaptively-adjust-spark-parameters

kylin.engine.spark.task-core-factor

Status
subtletrue
colourYellow
title

Medium

Integer

Status
subtletrue
titleProcess

3


kylin.engine.driver-memory-base

Status
subtletrue
colourYellow
title

Medium

Integer

Status
subtletrue
titleProcess

1024Auto adujst spark.driver.memory for Build Engine if kylin.engine.spark-conf.spark.driver.memory is not set.



Status
subtletrue
colourBlue
title4.0.0-alpha

Adaptively-adjust-spark-parameters
kylin.engine.driver-memory-strategy

Status
subtletrue
colourYellow
title

Medium

Array

Status
subtletrue
titleProcess

{"2", "20", "100"}



kylin.engine.driver-memory-maximum

Status
subtletrue
colourYellow
title

Medium

Integer

Status
subtletrue
titleProcess

4096


kylin.engine.persist-flattable-threshold

Status
subtletrue
colourYellow
title

Medium

Integer

Status
subtletrue
titleProcess

1If the number of cuboids which will be build from flat table is bigger than this threshold, the flat table will be persisted into $HDFS_WORKING_DIR/job_tmp/flat_table for saving more memory.

Status
subtletrue
colourBlue
title4.0.0-alpha


kylin.snapshot.parallel-build-timeout-seconds
Status
subtletrue
title

Status
subtletrue
colourRed
titleMAJOR


Status
subtletrue
titleProcess

3600
To improve the speed of snapshot build.


Status
subtletrue
colourBlue
title4.0.0-alpha


kylin.snapshot.parallel-build-enabled
Status
subtletrue
title

Status
subtletrue
colourRed
titleMAJOR

Boolean

Status
subtletrue
titleProcess

true





Status
subtletrue
titleProcess





kylin.spark-conf.auto.prior
Status
subtletrue
title

Status
subtletrue
titleMinor

Boolean

Status
subtletrue
titleProcess

true If need to adjust spark parameters adaptively.

Status
subtletrue
colourBlue
title4.0.0-alpha

Adaptively-adjust-spark-parameters
kylin.engine.submit-hadoop-conf-dir
Status
subtletrue
colourYellow
YES

Status
subtletrue
colourRed
titleMAJOR

String

Status
subtletrue
titleProcess

/etc/hadoop/conf
Set HADOOP_CONF_DIR for spark-submit.

Status
subtletrue
colourBlue
title4.0.0-alpha


kylin.storage.columnar.shard-size-mb

Status
subtletrue
colour

Yellow
titleYES

Status
subtletrue
colourRed
titleMAJOR

Integer

Status
subtletrue
colourRed
titleCube

128The size of each parquet partition file of cuboid

Status
subtletrue
colourBlue
title4.0.0-alpha

ShardBy
kylin.storage.columnar.shard-rowcount

Status
subtletrue
colour

Yellowtitle

Status
subtletrue
colourRed
titleMAJOR

Long

Status
subtletrue
colourRed
titleCube

2500000

The max rows of each parquet partition file of cuboid

kylin.storage.columnar.shard-countdistinct-rowcount

Status
subtletrue
colour

Yellow

YES

Status
subtletrue
colourRed
titleMAJOR

Long

Status
subtletrue
colourRed
titleCube

1000000The number rows of each parquet partition file of cuboid when the shard column is distinct column.

kylin.query.spark-engine.join-memory-fraction

Status
subtletrue
colourYellow
title

Medium

Double

Status
subtletrue
titleProcess

0.3Limit memory used by broadcast join.

Status
subtletrue
colourBlue
title4.0.0-alpha



...