Page tree
Skip to end of metadata
Go to start of metadata

This page describes the various strategies for executing queries in MetaModel.

Native vs greedy execution

Of particular interest is to specify in which cases MetaModel can delegate (aka "push down") query execution to a native query engine vs. having to execute the query in memory (often a greedy approach - Java code supplied by MetaModel).  

The table below documents the execution capability in specific modules of MetaModel. Each column represents a query type. The query types are:  

  • Plain FROM:
    • Simple queries of the form 'SELECT y FROM x'. Possible values: 
    • streaming: The dataset is implemented in a truly streaming fashion. 
    • paged: The dataset fetches pages/bulks of records. 
    • in-memory: The dataset has to consume ALL records into memory. This is ineffecient and may cause out of memory issues. 
  • Simple COUNT:
    • Queries of the form 'SELECT COUNT(*) FROM x'. Possible values: 
    • native: The module supports a effective native method of getting the count. Some modules also support additional criteria on COUNT queries, e.g. 'SELECT COUNT(*) FROM x WHERE z' which is marked as 'native (incl. WHERE)'. 
    • greedy: The module has to run through the dataset to do the counting. This is ineffecient but usually has little memory impact. 
  • Simple WHERE:
    • Are simple WHERE items being delegated natively, or are they evaluated client-side for each record? 
  • Primary key lookup:
    • Queries that look up records by their primary keys: SELECT y FROM x WHERE x.id = 42. 
  • Groups and aggregates:
    • Are GROUP BY and aggregation functions being delegated natively, or are they calculated in memory?

 

 Plain FROMSimple COUNTSimple WHEREPrimary key lookupGroups and aggregates
CSV(tick) Streaming

(warning) Greedy when exact.

(tick) Native when approximated.

(warning) Client-side(warning) No PK(minus) Greedy
JDBC(tick) Streaming(tick) Native (all variants)(tick) Native(tick) Native(tick) Native

Excel

(tick) Streaming XLSX

(warning) Greedy XLS

(tick) Native(warning) Client-side(warning) No PK(minus) Greedy
POJO(minus) In memory(tick) Native(warning) Client-side(warning) No PK(minus) Greedy
CouchDB(tick) Streaming(tick) Native(tick) Native(tick) Native(minus) Greedy
MongoDB(tick) Streaming(tick) Native(tick) Native(tick) Native(minus) Greedy
ElasticSearch(tick) Paged(tick) Native(tick) Native(tick) Native(minus) Greedy
HBase(tick) Streaming(tick) Native(warning) Client-side(tick) Native(minus) Greedy
Cassandra(tick) Paged(tick) Native(warning) Client-side(tick) Native(minus) Greedy
Amazon DynamoDB(tick) Paged(tick) Native(warning) Client-side(tick) Native(minus) Greedy
JSON(tick) Streaming(minus) Greedy(warning) Client-side(warning) No PK(minus) Greedy
XML

(tick) Streaming SAX

(warning) In-memory DOM

(minus) Greedy(warning) Client-side(minus) Greedy(minus) Greedy
Salesforce.com(tick) Paged(tick) Native(tick) Native(tick) Native(minus) Greedy
SugarCRM(tick) Paged(tick) Native(tick) Native(minus) Greedy(minus) Greedy
  • No labels