Spark SQL Guide

CREATE DATASOURCE TABLE

Description

The CREATE TABLE statement defines a new table using a Data Source.

Syntax

CREATE [ EXTERNAL ] TABLE [ IF NOT EXISTS ] table_identifier
    [ ( col_name1 col_type1 [ COMMENT col_comment1 ], ... ) ]
    USING data_source
    [ OPTIONS ( key1=val1, key2=val2, ... ) ]
    [ PARTITIONED BY ( col_name1, col_name2, ... ) ]
    [ CLUSTERED BY ( col_name3, col_name4, ... ) 
        [ SORTED BY ( col_name [ ASC | DESC ], ... ) ] 
        INTO num_buckets BUCKETS ]
    [ LOCATION path ]
    [ COMMENT table_comment ]
    [ TBLPROPERTIES ( key1=val1, key2=val2, ... ) ]
    [ DISTRIBUTED BY PARTITION ]
    [ [ LOCALLY ] ORDERED BY { ( write_order_field [ , ... ] ) | write_order_field [ , ... ] }
        | UNORDERED ]
    [ AS select_statement ]

write_order_field
    { col_name | transform ( { col_name | constant } [ , ... ] ) }
        [ ASC | DESC ] [ NULLS { FIRST | LAST } ]

Note that, the clauses between the USING clause and the AS SELECT clause can come in as any order. For example, you can write COMMENT table_comment after TBLPROPERTIES.

Parameters

Data Source Interaction

A Data Source table acts like a pointer to the underlying data source. For example, you can create a table “foo” in Spark which points to a table “bar” in MySQL using JDBC Data Source. When you read/write table “foo”, you actually read/write table “bar”.

In general CREATE TABLE is creating a “pointer”, and you need to make sure it points to something existing. An exception is file source such as parquet, json. If you don’t specify the LOCATION, Spark will create a default table location for you.

For CREATE TABLE AS SELECT with LOCATION, Spark throws analysis exceptions if the given location exists as a non-empty directory. If spark.sql.legacy.allowNonEmptyLocationInCTAS is set to true, Spark overwrites the underlying data source with the data of the input query, to make sure the table gets created contains exactly the same data as the input query.

Examples


--Use data source
CREATE TABLE student (id INT, name STRING, age INT) USING CSV;

-- External data source table. DROP TABLE keeps the files at the location.
CREATE EXTERNAL TABLE student (id INT, name STRING, age INT)
    USING parquet
    LOCATION '/tmp/student';

--Use data from another table
CREATE TABLE student_copy USING CSV
    AS SELECT * FROM student;
  
--Omit the USING clause, which uses the default data source (parquet by default)
CREATE TABLE student (id INT, name STRING, age INT);

--Use parquet data source with parquet storage options
--The columns 'id' and 'name' enable the bloom filter during writing parquet file,
--column 'age' does not enable
CREATE TABLE student_parquet(id INT, name STRING, age INT) USING PARQUET
    OPTIONS (
      'parquet.bloom.filter.enabled'='true',
      'parquet.bloom.filter.enabled#age'='false'
    );

--Specify table comment and properties
CREATE TABLE student (id INT, name STRING, age INT) USING CSV
    COMMENT 'this is a comment'
    TBLPROPERTIES ('foo'='bar');

--Specify table comment and properties with different clauses order
CREATE TABLE student (id INT, name STRING, age INT) USING CSV
    TBLPROPERTIES ('foo'='bar')
    COMMENT 'this is a comment';

--Create partitioned and bucketed table
CREATE TABLE student (id INT, name STRING, age INT)
    USING CSV
    PARTITIONED BY (age)
    CLUSTERED BY (Id) INTO 4 buckets;

--Create partitioned and bucketed table through CTAS
CREATE TABLE student_partition_bucket
    USING parquet
    PARTITIONED BY (age)
    CLUSTERED BY (id) INTO 4 buckets
    AS SELECT * FROM student;

--Create bucketed table through CTAS and CTE
CREATE TABLE student_bucket
    USING parquet
    CLUSTERED BY (id) INTO 4 buckets (
    WITH tmpTable AS (
        SELECT * FROM student WHERE id > 100
    )
    SELECT * FROM tmpTable
);