Package dev.vortex.spark

Class ArrowUtils

java.lang.Object
dev.vortex.spark.ArrowUtils
public final class ArrowUtils extends Object
Utility class for converting Arrow types to Spark SQL data types.

This class provides static methods to convert Arrow field definitions and type definitions into their corresponding Spark SQL DataType representations. The mapping matches Spark 4.1's own ArrowUtils, extended where Vortex produces types Spark's mapping does not cover:

  • the Arrow view types map to their logical Spark types: Utf8View to StringType, BinaryView to BinaryType, and ListView to ArrayType
  • timestamps of every unit map to TimestampType/TimestampNTZType; VortexArrowColumnVector normalizes non-microsecond values on read

The one intentional divergence from Spark 4.1 is Arrow's Time type: Spark's TimeType only exists in Spark 4.1+, and this module compiles a single source set against both Spark 3.5 and 4.1, so Time is rejected.

  • Method Summary

    Modifier and Type
    Method
    Description
    static org.apache.spark.sql.types.DataType
    fromArrowField(dev.vortex.relocated.org.apache.arrow.vector.types.pojo.Field field)
    Converts an Arrow Field to a Spark SQL DataType.
    static org.apache.spark.sql.types.DataType
    fromArrowType(dev.vortex.relocated.org.apache.arrow.vector.types.pojo.ArrowType dt)
    Converts an Arrow type to a Spark SQL DataType.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • fromArrowField

      public static org.apache.spark.sql.types.DataType fromArrowField(dev.vortex.relocated.org.apache.arrow.vector.types.pojo.Field field)
      Converts an Arrow Field to a Spark SQL DataType.

      This method handles nested types like structs, lists and maps by recursively converting their child fields. For non-nested types, it delegates to fromArrowType(ArrowType).

      Parameters:
      field - the Arrow field to convert
      Returns:
      the corresponding Spark SQL DataType
      Throws:
      UnsupportedOperationException - if the Arrow type is not supported

    • fromArrowType

      public static org.apache.spark.sql.types.DataType fromArrowType(dev.vortex.relocated.org.apache.arrow.vector.types.pojo.ArrowType dt)
      Converts an Arrow type to a Spark SQL DataType.

      This method maps non-nested Arrow types to their corresponding Spark SQL types, following Spark 4.1's own Arrow type mapping plus the view types (see the class documentation).

      Parameters:
      dt - the Arrow type to convert
      Returns:
      the corresponding Spark SQL DataType
      Throws:
      UnsupportedOperationException - if the Arrow type has no Spark representation