DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.

Apache Kylin

Space shortcuts

Apache Kylin : Analytical Data Warehouse for Big Data

Page tree

Versions Compared

Old Version 28

changes.mady.by.user Xiaoxiang Yu

Saved on

compared with

New Version 29

changes.mady.by.user Xiaoxiang Yu

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Advanced Tables - Table Plus
sortDescendingtrue
sortColumnPropertyImportance
autoNumberSorttrue
allowExporttrue
displayDataFiltertrue
sortIcontrue


Property
Importance
DatatypeConfiguration LevelDefaultDescriptionVersionReference
kylin.engine.spark.build-class-name

Status
subtletrue
titletrivial

String

Status
subtletrue
titleProcess

org.apache.kylin.engine.spark.job.CubeBuildJob
For developer only. The className use in spark-submit.

Status
subtletrue
colourBlue
title4.0.0-alpha


kylin.engine.spark.cluster-info-fetcher-class-name

Status
subtletrue
titletrivial

String

Status
subtletrue
titleProcess

org.apache.kylin.cluster.YarnInfoFetcher
For developer only. Fetch yarn information of spark job

Status
subtletrue
colourBlue
title4.0.0-alpha


kylin.engine.spark-conf.XXX

Status
subtletrue
titleMinor

String

Status
subtletrue
titleProcess

Null
  1. Before Kylin submit a cubing job, some major property(cores and memory) will be automatically adjusted adaptively. (if kylin.spark-conf.auto.prior was set to true).
  2. After auto adjust, spark conf will be overwrite by this property. If you want to set spark.driver.extraJavaOptions=-Dhdp.version=current, you can add follow line in kylin.properties:
kylin.engine.spark-conf.spark.driver.extraJavaOptions=-Dhdp.version=current
Spark configurations want to override for build job like "spark.driver.cores". If don't set these spark properties, kylin will automaticly adjust these properties before submitting build job. 
 Status
subtletrue
colourBlue
title4.0.0-alpha
Adaptively-adjust-spark-parameters
kylin.storage.provider
 Status
subtletrue
titletrivial
String
 Status
subtletrue
titleProcess
org.apache.kylin.common.storage.DefaultStorageProvider
The content summary objects returned by different cloud vendors are not the same, so need to provide targeted implementation.
You can refer to this to learn more : org.apache.kylin.common.storage.IStorageProvider
 Status
subtletrue
colourBlue
title4.0.0-alpha

kylin.engine.spark.merge-class-name
 Status
subtletrue
titletrivial
String
 Status
subtletrue
titleProcess
org.apache.kylin.engine.spark.job.CubeMergeJob
For developer only. The className use in spark-submit
 Status
subtletrue
colourBlue
title4.0.0-alpha

kylin.engine.spark.task-impact-instance-enabled
 Status
subtletrue
titletrivial
Boolean
 Status
subtletrue
titleProcess
true
 Status
subtletrue
colourYellow
titleUpdating
Check kylin.engine.spark.task-core-factor. If kylin.engine.spark.task-impact-instance-enabled is set to true and kylin.engine.spark-conf.spark.executor.instances is not set, Kylin will calculate spark.executor.instances for Build Engine.
 Status
subtletrue
colourBlue
title4.0.0-alpha
Adaptively-adjust-spark-parameters

kylin.engine.spark.task-core-factor
 Status
subtletrue
colourYellow
titleMedium
Integer
 Status
subtletrue
titleProcess
3
 Status
subtletrue
colourYellow
titleTo be updatedUpdating
 Status
subtletrue
colourBlue
title4.0.0-alpha

kylin.engine.driver-memory-base
 Status
subtletrue
colourYellow
titleMedium
Integer
 Status
subtletrue
titleProcess
1024Auto adujst spark.driver.memory for Build Engine if kylin.engine.spark-conf.spark.driver.memory is not set.



 Status
subtletrue
colourBlue
title4.0.0-alpha
Adaptively-adjust-spark-parameters
kylin.engine.driver-memory-strategy
 Status
subtletrue
colourYellow
titleMedium
Array
 Status
subtletrue
titleProcess
{"2", "20", "100"}
 Status
subtletrue
colourYellow
titleTo be updatedUpdating
 Status
subtletrue
colourBlue
title4.0.0-alpha

kylin.engine.driver-memory-maximum
 Status
subtletrue
colourYellow
titleMedium
Integer
 Status
subtletrue
titleProcess
4096
 Status
subtletrue
colourYellow
titleTo be updatedUpdating
 Status
subtletrue
colourBlue
title4.0.0-alpha

kylin.engine.persist-flattable-threshold
 Status
subtletrue
colourYellow
titleMedium
Integer
 Status
subtletrue
titleProcess
1If the number of cuboids which will be build from flat table is bigger than this threshold, the flat table will be persisted into $HDFS_WORKING_DIR/job_tmp/flat_table for saving more memory.
 Status
subtletrue
colourBlue
title4.0.0-alpha

kylin.snapshot.parallel-build-timeout-seconds
 Status
subtletrue
colourRed
titleMAJOR
Integer
 Status
subtletrue
titleProcess
3600
To improve the speed of snapshot build.

 Status
subtletrue
colourBlue
title4.0.0-alpha

kylin.snapshot.parallel-build-enabled
 Status
subtletrue
colourRed
titleMAJOR
Boolean
 Status
subtletrue
titleProcess
true
 Status
subtletrue
colourYellow
titleTo be updatedUpdating










kylin.spark-conf.auto.prior
 Status
subtletrue
colourGreen
titleMinor
Boolean
 Status
subtletrue
titleProcess
true Enable adjust spark parameters adaptively.
 Status
subtletrue
colourBlue
title4.0.0-alpha
Adaptively-adjust-spark-parameters
kylin.engine.submit-hadoop-conf-dir
 Status
subtletrue
colourRed
titleMAJOR
String
 Status
subtletrue
titleProcess
/etc/hadoop/conf
Set HADOOP_CONF_DIR for spark-submit.
 Status
subtletrue
colourBlue
title4.0.0-alpha

kylin.storage.columnar.shard-size-mb
 Status
subtletrue
colourRed
titleMAJOR
Integer
 Status
subtletrue
colourRed
titleCube
128
The max size of pre-calcualted cuboid parquet file.
 Status
subtletrue
colourBlue
title4.0.0-alpha
ShardBy
kylin.storage.columnar.shard-rowcount
 Status
subtletrue
colourRed
titleMAJOR
Long
 Status
subtletrue
colourRed
titleCube
2500000
The max rows of pre-calcualted cuboid parquet file.
 Status
subtletrue
colourBlue
title4.0.0-alpha

kylin.storage.columnar.shard-countdistinct-rowcount
 Status
subtletrue
colourRed
titleMAJOR
Long
 Status
subtletrue
colourRed
titleCube
1000000The max rows of pre-calcualted cuboid parquet file when cuboid has bitmap measure. (When cuboid has BItmap, it is large.)
 Status
subtletrue
colourBlue
title4.0.0-alpha

kylin.query.spark-engine.join-memory-fraction
 Status
subtletrue
colourYellow
titleMedium
Double
 Status
subtletrue
titleProcess
0.3Limit memory used by broadcast join of Sparder. (Broadcast join cause unstable.)
 Status
subtletrue
colourBlue
title4.0.0-alpha


...
File NameContentComment
cubing_detect_items.json

sampling_detect_items.json


count_distinct.json
true/false
resource_paths.json