[geomesa-dev] [JIRA] (GEOMESA-2610) timestamp columns are not supported

["Chris Slater - NOAA Affiliate (JIRA)" <jira@xxxxxxxxxxxxxxxxxxxxx>]

Title: Message Title

Chris Slater - NOAA Affiliate created an issue

GeoMesa / GEOMESA-2610

timestamp columns are not supported with Spark SQL

Issue Type:	Bug
Affects Versions:	2.3.0
Assignee:	Unassigned
Created:	26/Apr/19 5:18 PM
Priority:	Major
Reporter:	Chris Slater - NOAA Affiliate

According to https://www.geomesa.org/documentation/user/datastores/attributes.html?highlight=types, java.sql.Timestamp is the binding class for a Timestamp type.

I am using the following schema:

cruise_name:String,frequency_khz:Integer,ping_index:Integer,distance_gps:Double,distance_vl:Double,timestamp:Timestamp,*ping_location:Point:srid=4326,depth_start:Double,depth_stop:Double,range_start:Integer,range_stop:Integer,data:List[Double]

I am using the following code:

    SparkSession spark = SparkSession.builder()
        .appName(APP_NAME)
        .config("spark.sql.crossJoin.enabled", "true")
        .master("local[*]")
        .getOrCreate();
    SQLTypes.init(spark.sqlContext());

    Dataset<Row> df = spark.read()
        .format("geomesa")
        .options(datastoreParams.getParams())
        .option("geomesa.feature", "myfeature")
        .load();
    df.createOrReplaceTempView("myfeature");

    df.explain();
    df.printSchema();

The result of this is:

root
 |-- __fid__: string (nullable = false)
 |-- cruise_name: string (nullable = true)
 |-- frequency_khz: integer (nullable = true)
 |-- ping_index: integer (nullable = true)
 |-- distance_gps: double (nullable = true)
 |-- distance_vl: double (nullable = true)
 |-- ping_location: point (nullable = true)
 |-- depth_start: double (nullable = true)
 |-- depth_stop: double (nullable = true)
 |-- range_start: integer (nullable = true)
 |-- range_stop: integer (nullable = true)

Notice that timestamp is missing.

In the source code for org.locationtech.geomesa.spark.GeoMesaSparkSQL, there is this:

private def ad2field(ad: AttributeDescriptor): Option[StructField] = {
  import java.{lang => jl}
  val dt = ad.getType.getBinding match {
    case t if t == classOf[jl.Double]                       => DataTypes.DoubleType
    case t if t == classOf[jl.Float]                        => DataTypes.FloatType
    case t if t == classOf[jl.Integer]                      => DataTypes.IntegerType
    case t if t == classOf[jl.String]                       => DataTypes.StringType
    case t if t == classOf[jl.Boolean]                      => DataTypes.BooleanType
    case t if t == classOf[jl.Long]                         => DataTypes.LongType
    case t if t == classOf[java.util.Date]                  => DataTypes.TimestampType

    case t if t == classOf[org.locationtech.jts.geom.Point]            => JTSTypes.PointTypeInstance
    case t if t == classOf[org.locationtech.jts.geom.MultiPoint]       => JTSTypes.MultiPointTypeInstance
    case t if t == classOf[org.locationtech.jts.geom.LineString]       => JTSTypes.LineStringTypeInstance
    case t if t == classOf[org.locationtech.jts.geom.MultiLineString]  => JTSTypes.MultiLineStringTypeInstance
    case t if t == classOf[org.locationtech.jts.geom.Polygon]          => JTSTypes.PolygonTypeInstance
    case t if t == classOf[org.locationtech.jts.geom.MultiPolygon]     => JTSTypes.MultipolygonTypeInstance

    case t if      classOf[Geometry].isAssignableFrom(t)    => JTSTypes.GeometryTypeInstance

    // NB:  List and Map types are not supported.
    case _                                                  => null
  }
  Option(dt).map(StructField(ad.getLocalName, _))
}

java.sql.Timestamp is not listed as an option.

Add Comment

Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS
This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100100-sha1:1384dbb)