7. Apache Griffin DSL Guidance

source.id = target.id AND source.name = target.name AND source.age = target.age

SELECT source.country, source.id.count() AS count, source.age.avg() AS avg_age FROM source GROUP BY source.country SORT BY count DESC LIMIT 5

-- literal --
<literal> ::= <literal-string> | <literal-number> | <literal-time> | <literal-boolean> | <literal-null> | <literal-nan>
<literal-string> ::= <any-string>
<literal-number> ::= <integer> | <double>
<literal-time> ::= <integer> ("d"|"h"|"m"|"s"|"ms")
<literal-boolean> ::= true | false
<literal-null> ::= null
<literal-nan> ::= nan

-- selection --
<selection> ::= <selection-head> [ <field-sel> | <index-sel> | <function-sel> ]* [<as-alias>]?
<selection-head> ::= ("data source name registered") | <function> | <field-name> | <all-selection>
<field-sel> ::= "." <field-name> | "[" <quote-field-name> "]"
<index-sel> ::= "[" <arg> "]"
<function-sel> ::= "." <function-name> "(" [<arg>]? [, <arg>]* ")"
<arg> ::= <math-expr>

-- as alias --
<as-alias> ::= <as> <field-name>

-- math expr --
<math-factor> ::= <literal> | <function> | <selection> | "(" <math-expr> ")" [<as-alias>]?
<unary-math-expr> ::= [<unary-opr>]* <math-factor>
<binary-math-expr> ::= <unary-math-expr> [<binary-opr> <unary-math-expr>]+
<math-expr> ::= <binary-math-expr>

-- logical expr --
<in-expr> ::= <math-expr> [<not>]? <in> <range-expr>
<between-expr> ::= <math-expr> [<not>]? <between> (<math-expr> <and> <math-expr> | <range-expr>)
<range-expr> ::= "(" [<math-expr>]? [, <math-expr>]+ ")"
<like-expr> ::= <math-expr> [<not>]? <like> <math-expr>
<is-null-expr> ::= <math-expr> <is> [<not>]? <null>
<is-nan-expr> ::= <math-expr> <is> [<not>]? <nan>
<logical-factor> ::= <math-expr> | <in-expr> | <between-expr> | <like-expr> | <is-null-expr> | <is-nan-expr> | "(" <logical-expr> ")" [<as-alias>]?
<unary-logical-expr> ::= [<unary-logical-opr>]* <logical-factor>
<binary-logical-expr> ::= <unary-logical-expr> [<binary-logical-opr> <unary-logical-expr>]+
<logical-expr> ::= <binary-logical-expr>

-- expression --
<expr> = <math-expr> | <logical-expr>

-- function expr --
<function> ::= <function-name> "(" [<arg>] [, <arg>]+ ")" [<as-alias>]?
<function-name> ::= ("function name registered")
<arg> ::= <expr>

-- clauses --
<select-clause> = <expr> [, <expr>]*
<where-clause> = <where> <expr>
<from-clause> = <from> ("data source name registered")
<having-clause> = <having> <expr>
<groupby-clause> = <group> <by> <expr> [ <having-clause> ]?
<orderby-item> = <expr> [ <DESC> ]?
<orderby-clause> = <order> <by> <orderby-item> [ , <orderby-item> ]*
<limit-clause> = <limit> <expr>

-- combined clauses --
<combined-clauses> = <select-clause> [ <from-clause> ]+ [ <where-clause> ]+ [ <groupby-clause> ]+ [ <orderby-clause> ]+ [ <limit-clause> ]+

- null, nan, true, false
- not, and, or
- in, between, like, is
- select, distinct, from, as, where, group, by, having, order, desc, asc, limit

- !, &&, ||, =, !=, <, >, <=, >=, <>
- +, -, *, /, %
- (, )
- ., [, ]

- string: any string surrounded with a pair of " or ', with escape charactor \ if any request.  
	e.g. "test", 'string 1', "hello \" world \" "
- number: double or integer number.  
	e.g. 123, 33.5
- time: a integer with unit in a string, will be translated to a integer number in millisecond.  
	e.g. 3d, 5h, 4ms
- boolean: boolean value literally.  
	e.g. true, false

- selection head: data source name.
	e.g. source, target, `my table name`
- all field selection: * or with data source name ahead.
	e.g. *, source.*, target.*
- field selection: field name or with data source name ahead.
	e.g. source.age, target.name, user_id
- index selection: interget between square brackets "[]" with field name ahead.
	e.g. source.attributes[3]
- function selection: function name with brackets "()", with field name ahead or not.
	e.g. count(*), *.count(), source.user_id.count(), max(source.age)
- alias: declare an alias after a selection.
	e.g. source.user_id as id, target.user_name as name

- math factor: literal or function or selection or math exression with brackets.
	e.g. 123, max(1, 2, 3, 4), source.age, (source.age + 13)
- unary math expression: unary math operator with factor.
	e.g. -(100 - source.score)
- binary math expression: math factors with binary math operators.
	e.g. source.age + 13, score * 2 + ratio

- in: in clause like sql.
	e.g. source.country in ("USA", "CHN", "RSA")
- between: between clause like sql.
	e.g. source.age between 3 and 30, source.age between (3, 30)
- like: like clause like sql.
	e.g. source.name like "%abc%"
- is null: is null operator like sql.
	e.g. source.desc is not null
- is nan: check if the value is not a number, the syntax like is null
	e.g. source.age is not nan
- logical factor: math expression or logical expressions above or other logical expressions with brackets.
	e.g. (source.user_id = target.user_id AND source.age > target.age)
- unary logical expression: unary logical operator with factor.
	e.g. NOT source.has_data, !(source.age = target.age)
- binary logical expression: logical factors with binary logical operators, including and, or and comparison operators.
	e.g. source.age = target.age OR source.ticket = target.tck

- expression: logical expression and math expression.

- argument: expression.
- function: function name with arguments between brackets.
	e.g. max(source.age, target.age), count(*)

- select clause: the result columns like sql select clause, we can ignore the word "select" in Griffin DSL.
	e.g. select user_id.count(), age.max() as max, source.user_id.count() as cnt, source.age.min()
- from clause: the table name like sql from clause, in which the data source name must be one of data source names or the output table name of the former rule steps, we can ignore this clause by configoring the data source name.
	e.g. from source, from `target`
- where clause: the filter condition like sql where clause, optional.
	e.g. where source.id = target.id and source.age = target.age
- group-by clause: like the group-by clause in sql, optional. Optional having clause could be following.
	e.g. group by cntry, group by gender having count(*) > 50
- order-by clause: like the order-by clause, optional.
	e.g. order by name, order by first_name desc, age asc
- limit clause: like the limit clause in sql, optional.
	e.g. limit 5

- Accuracy rule expression in Griffin DSL is a logical expression, telling the mapping relation between data sources.
	e.g. source.id = target.id and source.name = target.name and source.age between (target.age, target.age + 5

- Profiling rule expression in Griffin DSL is a sql-like expression, with select clause ahead, following optional from clause, where clause, group-by clause, order-by clause, limit clause in order.
	e.g. source.gender, source.id.count() where source.age > 20 group by source.gender, select country, max(age), min(age), count(*) as cnt from source group by country order by cnt desc limit 5

- Uniqueness rule expression in Griffin DSL is a list of selection expressions separated by comma, indicates the duplicate columns to measure.
	e.g. name, (age + 1) as next_age

- Timeliness rule expression in Griffin DSL is a list of selection expressions separated by comma, indicates the input time and output time (calculate time as default if not set).
	e.g. ts

{
	"dsl.type": "spark-sql",
	"name": "source",
	"rule": "SELECT count(id) AS cnt, max(timestamp) AS fresh_time FROM source"
}

{
	"dsl.type": "df-opr",
	"name": "ext_source",
	"rule": "from_json",
	"details": {
		"df.name": "json_source"
	}
}