Package dev.vortex.spark

Class VortexCatalog

java.lang.Object
dev.vortex.spark.VortexCatalog
All Implemented Interfaces:
org.apache.spark.sql.connector.catalog.CatalogPlugin, org.apache.spark.sql.connector.catalog.TableCatalog
public final class VortexCatalog extends Object implements org.apache.spark.sql.connector.catalog.TableCatalog
A path-based Spark catalog for querying Vortex files directly from SQL.

Spark only supports SELECT * FROM format.`path` syntax for built-in file formats, so this catalog provides the equivalent for Vortex. Register it under the name vortex: 

spark.sql.catalog.vortex=dev.vortex.spark.VortexCatalog

then query a Vortex file, or a directory of Vortex files, directly by path: 

SELECT * FROM vortex.`/path/to/data`;

The table identifier must look like a path — contain a / — and resolves to the same table a spark.read.format("vortex") load of that path would produce, so reads, writes (INSERT INTO vortex.`/path/to/data`), and pushdown all behave identically. The catalog holds no state and supports no DDL.

  • Field Summary

    Fields inherited from interface org.apache.spark.sql.connector.catalog.TableCatalog

    OPTION_PREFIX, PROP_COLLATION, PROP_COMMENT, PROP_EXTERNAL, PROP_IS_MANAGED_LOCATION, PROP_LOCATION, PROP_OWNER, PROP_PROVIDER, PROP_TABLE_TYPE

  • Constructor Summary

    Constructors
    Constructor
    Description
    VortexCatalog()
    Creates a new catalog instance.

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.spark.sql.connector.catalog.Table
    alterTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.connector.catalog.TableChange... changes)
    Unsupported: this catalog holds no table metadata to alter.
    org.apache.spark.sql.connector.catalog.Table
    createTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitions, Map<String,String> properties)
    Unsupported: tables are addressed by path, create them by writing data with the vortex format.
    boolean
    dropTable(org.apache.spark.sql.connector.catalog.Identifier ident)
    Unsupported: this catalog never drops data, returns false.
    void
    initialize(String name, org.apache.spark.sql.util.CaseInsensitiveStringMap options)
     
    org.apache.spark.sql.connector.catalog.Identifier[]
    listTables(String[] namespace)
    Returns no identifiers: this catalog holds no state, tables are addressed by path.
    org.apache.spark.sql.connector.catalog.Table
    loadTable(org.apache.spark.sql.connector.catalog.Identifier ident)
    Loads the Vortex file or directory of Vortex files at the path given by the identifier name.
    String
    name()
     
    void
    renameTable(org.apache.spark.sql.connector.catalog.Identifier oldIdent, org.apache.spark.sql.connector.catalog.Identifier newIdent)
    Unsupported: this catalog holds no table metadata to rename.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.spark.sql.connector.catalog.CatalogPlugin

    defaultNamespace

    Methods inherited from interface org.apache.spark.sql.connector.catalog.TableCatalog

    capabilities, createTable, createTable, invalidateTable, listTableSummaries, loadTable, loadTable, loadTable, purgeTable, tableExists, useNullableQuerySchema

  • Constructor Details

    • VortexCatalog

      public VortexCatalog()
      Creates a new catalog instance.

      This no-argument constructor is required for Spark to instantiate the catalog through reflection from the spark.sql.catalog.<name> configuration.

  • Method Details

    • initialize

      public void initialize(String name, org.apache.spark.sql.util.CaseInsensitiveStringMap options)
      Specified by:
      initialize in interface org.apache.spark.sql.connector.catalog.CatalogPlugin

    • name

      public String name()
      Specified by:
      name in interface org.apache.spark.sql.connector.catalog.CatalogPlugin

    • listTables

      public org.apache.spark.sql.connector.catalog.Identifier[] listTables(String[] namespace)
      Returns no identifiers: this catalog holds no state, tables are addressed by path.
      Specified by:
      listTables in interface org.apache.spark.sql.connector.catalog.TableCatalog
      Parameters:
      namespace - the namespace to list, ignored
      Returns:
      an empty array

    • loadTable

      public org.apache.spark.sql.connector.catalog.Table loadTable(org.apache.spark.sql.connector.catalog.Identifier ident) throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException
      Loads the Vortex file or directory of Vortex files at the path given by the identifier name.
      Specified by:
      loadTable in interface org.apache.spark.sql.connector.catalog.TableCatalog
      Parameters:
      ident - identifier whose name is a filesystem path or URL, e.g. vortex.`/path/to/data`
      Returns:
      a table backed by the Vortex files at the path
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchTableException - if the identifier does not look like a path, or the path cannot be read

    • createTable

      public org.apache.spark.sql.connector.catalog.Table createTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitions, Map<String,String> properties)
      Unsupported: tables are addressed by path, create them by writing data with the vortex format.
      Specified by:
      createTable in interface org.apache.spark.sql.connector.catalog.TableCatalog

    • alterTable

      public org.apache.spark.sql.connector.catalog.Table alterTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.connector.catalog.TableChange... changes)
      Unsupported: this catalog holds no table metadata to alter.
      Specified by:
      alterTable in interface org.apache.spark.sql.connector.catalog.TableCatalog

    • dropTable

      public boolean dropTable(org.apache.spark.sql.connector.catalog.Identifier ident)
      Unsupported: this catalog never drops data, returns false.
      Specified by:
      dropTable in interface org.apache.spark.sql.connector.catalog.TableCatalog

    • renameTable

      public void renameTable(org.apache.spark.sql.connector.catalog.Identifier oldIdent, org.apache.spark.sql.connector.catalog.Identifier newIdent)
      Unsupported: this catalog holds no table metadata to rename.
      Specified by:
      renameTable in interface org.apache.spark.sql.connector.catalog.TableCatalog