Hi – my ingestion fails when bringing in a date attribute. The date and time is in UTC, ISO-8601 format, example: "2014-04-01T08:33:35.000Z". I have successfully set up the .sft and .convert and the ingest wiil work when I format the
date type as a “string”. However, the features fail to ingest when I do:
.convert:
{ name = "tripid", transform = "$1::string" }
{ name = "wp", transform = "$2::int" }
{ name = "capdate", transform = "dateTime($3)" }
{ name = "lon", transform = "$4::double" }
{ name = "lat", transform = "$5::double" }
And .sft:
{ name = "tripid", type = "String", index = true }
{ name = "wp", type = "Integer", index = true }
{ name = "capdate", type = "Date", index = false }
{ name = "lon", type = "Double", index = false }
{ name = "lat", type = "Double", index = false }
Sample date times are
"2016-01-18 18:35:49.33+00"
"2016-02-22 01:24:00+00"
"2016-02-29 22:24:21+00"
"2016-03-21 13:21:47.173+00"
"2016-01-18 14:36:33+00"
"2016-02-22 02:28:26.98+00"
"2016-02-22 15:29:58.15+00"
I am ingesting as CSV – skip line 1 from a S3 bucket. I don’t know if it is because the millis are sometimes present and sometimes not (though postgres does not seem bothered by this). Any hints?
Thanks,
Byron