Read a Particular Element of Json File Spark Scala

Update (18.11.2015): added spray-json-shapeless library
Update (06.eleven.xv): added circe library

Some time ago I wrote a post on relational database access in Scala since I was looking for a library and there were many of them available, making it hard to brand a choice. It turns out that the situation is similar if not worse when it comes to JSON libraries in Scala.

In that location are just plenty of them. You take no idea. (me neither until I wrote this post)

The following is an endeavour to provide a quick overview at how a subset of the libraries I found does a few of the most common things 1 would likely need to do in regard to JSON:

• parsing it from a raw string
• browsing the AST
• edifice an AST
• mapping to a case class

There are of class enough more valid use-cases but this post is not going to cover those.

Let's go started!

Scala JSON libraries

play-json

The Play Framework comes with a JSON library that covers about of the common use-cases one would encounter when edifice spider web applications:

Parsing raw JSON

scala > import play . api . libs . json . _ import play . api . libs . json . _ scala > val rawJson = "" "{" hi ": " earth ", " age ": 42}" "" rawJson : String = { "hello" : "earth" , "age" : 42 } scala > Json . parse ( rawJson ) res0 : play . api . libs . json . JsValue = { "hello" : "world" , "age" : 42 }

Browsing the AST

scala > ( res0 \ "hello" ) . as [ String ] res1 : String = world

Edifice a JSON AST

scala > Json . obj ( "howdy" -> "world" , "age" -> 42 ) res2 : play . api . libs . json . JsObject = { "hello" : "world" , "age" : 42 }

Mapping to a instance class

scala > case class Model ( hello : String , age : Int ) divers class Model scala > implicit val modelFormat = Json . format [ Model ] modelFormat : play . api . libs . json . OFormat [ Model ] = play . api . libs . json . OFormat $ $ anon $ 1 @ 81116d scala > Json . fromJson [ Model ] ( res0 ) res3 : play . api . libs . json . JsResult [ Model ] = JsSuccess ( Model ( earth , 42 ) , ) scala > res3 . go res4 : Model = Model ( earth , 42 ) scala > Json . toJson ( Model ( "bar" , 23 ) ) res5 : play . api . libs . json . JsValue = { "how-do-you-do" : "bar" , "age" : 23 }

lift-json

elevator-json is the JSON library of the Lift framework. Information technology is one of the oldest i out there if I'm not mistaken.

Parsing raw JSON

scala > import net . liftweb . json . _ import net . liftweb . json . _ scala > val rawJson = "" "{" howdy ": " world ", " historic period ": 42, " nested ": { " deeper ": { " treasure ": true }}}" "" rawJson : String = { "hello" : "earth" , "historic period" : 42 , "nested" : { "deeper" : { "treasure" : true } } } scala > parse ( rawJson ) res0 : net . liftweb . json . JValue = JObject ( List ( JField ( hi , JString ( world ) ) , JField ( age , JInt ( 42 ) ) , JField ( nested , JObject ( List ( JField ( deeper , JObject ( List ( JField ( treasure , JBool ( true ) ) ) ) ) ) ) ) ) )

Browsing the AST

scala > res0 \ "nested" \ "deeper" \ "treasure" res1 : internet . liftweb . json . JsonAST . JValue = JBool ( truthful )

Building a JSON AST

scala > import net . liftweb . json . JsonDSL . _ import net . liftweb . json . JsonDSL . _ scala > ( "hello" -> "world" ) ~ ( "age" -> 42 ) res2 : net . liftweb . json . JsonAST . JObject = JObject ( List ( JField ( hello , JString ( world ) ) , JField ( age , JInt ( 42 ) ) ) )

Mapping to a case class

object LiftJsonExample { def master ( args : Array [ String ] ) : Unit = { import internet . liftweb . json . _ implicit val formats = DefaultFormats example class Model ( hello : Cord , historic period : Int ) val rawJson = "" "{" hello ": " world ", " age ": 42}" "" println ( parse ( rawJson ) . excerpt [ Model ] ) } }

spray-json

spray rols its own JSON library that focuses on working with case classes:

Parsing raw JSON

scala > import spray . json . _ import spray . json . _ scala > import DefaultJsonProtocol . _ import DefaultJsonProtocol . _ scala > val rawJson = "" "{" hello ": " world ", " age ": 42}" "" rawJson : String = { "hello" : "world" , "age" : 42 } scala > rawJson . parseJson res0 : spray . json . JsValue = { "hullo" : "earth" , "age" : 42 }

Browsing the AST

No can do?

Building a JSON AST

No tin can do?

Mapping to a case class

scala > case course Model ( how-do-you-do : String , historic period : Int ) defined class Model scala > implicit val modelFormat = jsonFormat2 ( Model ) modelFormat : spray . json . RootJsonFormat [ Model ] = spray . json . ProductFormatsInstances $ $ betimes $ 2 @ 7bc880f8 scala > res1 . convertTo [ Model ] res4 : Model = Model ( globe , 42 )

spray-json-shapeless

spray-json-shapeless derives spray-json's JsonFormat-southward automatically using shapeless (no demand to ascertain an implicit JsonFormat

Mapping to a instance grade

scala > import spray . json . _ import spray . json . _ scala > import fommil . sjs . FamilyFormats . _ import fommil . sjs . FamilyFormats . _ scala > example class Model ( hullo : String , age : Int ) defined grade Model scala > Model ( "hello" , 42 ) . toJson res0 : spray . json . JsValue = { "hello" : "hello" , "age" : 42 }

This is quite useful, it removes the boilerplate formats hanging effectually

json4s

json4s is a bit like slf4j in the sense that it tries to unite all kind of rogue libraries serving the aforementioned purpose past providing a common interface. But not every library uses it, which means that chances are loftier that your project will comprise json4s in improver to some other (few) Scala JSON libraries. Hopefully information technology will i day succeed with its slogan "One AST to rule them all".

json4s has its roots in lift-json so this will look familiar:

Parsing raw JSON

scala > import org . json4s . _ import org . json4s . _ scala > import org . json4s . native . JsonMethods . _ import org . json4s . native . JsonMethods . _ scala > val rawJson = "" "{" hullo ": " world ", " historic period ": 42}" "" rawJson : String = { "hi" : "world" , "age" : 42 } scala > parse ( rawJson ) res0 : org . json4s . JValue = JObject ( List ( ( hello , JString ( earth ) ) , ( age , JInt ( 42 ) ) ) )

Browsing the AST

scala > res0 \ "hello" res1 : org . json4s . JValue = JString ( world )

Building a JSON AST tree

scala > ( "hello" -> "world" ) ~ ( "age" -> 42 ) res2 : org . json4s . JsonAST . JObject = JObject ( List ( ( hello , JString ( world ) ) , ( age , JInt ( 42 ) ) ) )

Mapping to a instance grade

object Json4sExample { def chief ( args : Array [ String ] ) : Unit = { import org . json4s . _ import org . json4s . native . JsonMethods . _ implicit val formats = DefaultFormats case grade Model ( hello : String , age : Int ) val rawJson = "" "{" hello ": " world ", " historic period ": 42}" "" println ( parse ( rawJson ) . extract [ Model ] ) } }

argonaut

Argonaut promotes "purely functional JSON in Scala". It uses scalaz under the hood and

Parsing raw JSON

import scalaz . _ , Scalaz . _ import scalaz . _ import Scalaz . _ scala > import argonaut . _ , Argonaut . _ import argonaut . _ import Argonaut . _ scala > val rawJson = "" "{" hello ": " world ", " historic period ": 42, " nested ": { " deeper ": { " treasure ": true }}}" "" rawJson : String = { "how-do-you-do" : "world" , "age" : 42 , "nested" : { "deeper" : { "treasure" : truthful } } } scala > rawJson . parseOption res0 : Choice [ argonaut . Json ] = Some ( { "hi" : "globe" , "age" : 42 , "nested" : { "deeper" : { "treasure" : true } } } )

Browsing the AST

At that place are several mechanisms available, let's use a lense. Those are funky:

scala > val ohMyLens = jObjectPL >=> jsonObjectPL ( "nested" ) >=> jObjectPL >=> jsonObjectPL ( "deeper" ) >=> jObjectPL >=> jsonObjectPL ( "treasure" ) >=> jBoolPL ohMyLens : scalaz . PLensFamily [ argonaut . Json , argonaut . Json , Boolean , Boolean ] = scalaz . PLensFamilyFunctions $ $ betimes $ ii @ 8c894ab scala > ohMyLens . get ( res0 . get ) res1 : Option [ Boolean ] = Some ( true )

Building a JSON AST tree

scala > ( "hi" , jString ( "earth" ) ) -> : ( "age" , jNumber ( 42 ) ) -> : jEmptyObject res2 : argonaut . Json = { "age" : 42 , "hello" : "world" }

Mapping to a example class

scala > case form Model ( hello : String , age : Int ) divers course Model scala > implicit def ModelCodecJson : CodecJson [ Model ] = casecodec2 ( Model . apply , Model . unapply ) ( "hullo" , "age" ) ModelCodecJson : argonaut . CodecJson [ Model ] scala > rawJson . decodeOption [ Model ] res3 : Option [ Model ] = Some ( Model ( globe , 42 ) )

circe

Circe is a fork of Argonaut that uses cats instead of scalaz and uses shapeless to generate codecs.

Parsing raw JSON

1 ii 3 4 5 6 vii 8 9 ten 11 12 13 14 15 16 17 18 19

scala > import io . circe . _ , io . circe . generic . automobile . _ , io . circe . parse . _ , io . circe . syntax . _ import io . circe . _ import io . circe . generic . car . _ import io . circe . parse . _ import io . circe . syntax . _ scala > val rawJson = "" "{" hullo ": " world ", " age ": 42, " nested ": { " deeper ": { " treasure ": true }}}" "" rawJson : String = { "hello" : "globe" , "historic period" : 42 , "nested" : { "deeper" : { "treasure" : true } } } scala > parse ( rawJson ) . getOrElse ( Json . empty ) res0 : io . circe . Json = { "hello" : "globe" , "historic period" : 42 , "nested" : { "deeper" : { "treasure" : true } } }

Browsing the AST

And then far in that location'south only support for cursors which let you movement around the JSON tree and do changes if y'all would like, not for Lenses yet:

scala > val cursor = res0 . cursor cursor : io . circe . Cursor = CJson ( { "hello" : "world" , "age" : 42 , "nested" : { "deeper" : { "treasure" : true } } } ) scala > for { | nested <- cursor . downField ( "nested" ) | deeper <- nested . downField ( "deeper" ) | treasure <- deeper . downField ( "treasure" ) | } yield treasure . as [ Boolean ] res1 : Option [ io . circe . Decoder . Consequence [ Boolean ] ] = Some ( Right ( true ) )

Mapping to a example class hierarchy

I didn't get this to work with the instance below. The example in the documentation works though so here is that 1:

1 2 3 4 five 6 7 8 ix 10 eleven 12 xiii 14 15 16 17 eighteen

scala > sealed trait Foo defined trait Foo scala > example class Bar ( xs : List [ Cord ] ) extends Foo defined class Bar scala > example class Qux ( i : Int , d : Option [ Double ] ) extends Foo defined form Qux scala > val foo : Foo = Qux ( thirteen , Some ( 14.0 ) ) foo : Foo = Qux ( 13 , Some ( fourteen.0 ) ) scala > foo . asJson . noSpaces res0 : String = { "Qux" : { "d" : 14.0 , "i" : 13 } } scala > decode [ Foo ] ( foo . asJson . spaces4 ) res1 : cats . information . Xor [ io . circe . Error , Foo ] = Correct ( Qux ( 13 , Some ( 14.0 ) ) )

sphere-json

The sphere-json library focuses on providing an like shooting fish in a barrel way to get de/serializers for entire families of case classes. This is actually useful when working with whatsoever kind of protocol-related system. I am using it in a CQRS system where commands and events are travelling dorsum and forth between the server and a Javascript UI. Instead of having to define a codec for each one of my case classes, I simply take them extend a common abstract class and let the library take care of the balance. Watch for yourselves:

Mapping to a example form bureaucracy

1 ii 3 4 v 6 7 8 9 10 11 12 thirteen 14 15 xvi 17 18 19 20 21 22 23 24 25 26 27 28

import io . sphere . json . generic . _ import io . sphere . json . _ object Letters { sealed abstract class Message instance class Hullo ( hi : String ) extends Message case course Age ( age : Int ) extends Message } object SphereJsonExample { import Messages . _ val rawHello = "" "{ " howdy ": " world ", " type ": " Hello " }" "" val rawAge = "" "{ " historic period ": 42, " type ": " Age " }" "" implicit val allYourJson = deriveJSON [ Bulletin ] def magic ( json : Cord ) = fromJSON [ Message ] ( json ) . fold ( neglect => println ( "Oh noes: " + fail ) , hi => println ( hi ) ) def main ( args : Array [ String ] ) : Unit of measurement = { magic ( rawHello ) magic ( rawAge ) } }

jawn

jawn is a library that focuses on speed. It defines a lightweight AST and is compatible with all kind of other ASTs that we have seen here.

Parsing raw JSON

scala > import jawn . _ import jawn . _ scala > import jawn . ast . _ import jawn . ast . _ scala > val rawJson = "" "{" hi ": " earth ", " age ": 42}" "" rawJson : String = { "howdy" : "world" , "age" : 42 } scala > jawn . ast . JParser . parseFromString ( rawJson ) . get res0 : jawn . ast . JValue = { "age" : 42 , "hi" : "earth" }

rapture

rapture'south json library is the ultimate Scala JSON library. It doesn't really do anything with JSON itself, instead, it abstracts over the following JSON libraries (which it calls backends):

• Argonaut
• Jackson
• Jawn
• JSON4S
• Lift
• Play
• Scala standard library JSON
• Spray

Parsing raw JSON

scala > import rapture . json . _ scala > import rapture . json . jsonBackends . play . _ import rapture . json . jsonBackends . play . _ scala > Json . parse ( rawJson ) res2 : rapture . json . Json = { "howdy" : "world" , "age" : 42 }

Browsing the AST

Rapture is using Scala'south Dynamic trait, which makes this fun:

scala > res2 . hello . as [ Cord ] res3 : String = world

Building a JSON AST tree

scala > json "" "{ " how-do-you-do ": " world ", " historic period ": 42}" "" res6 : rapture . json . Json = { "hello" : "globe" , "historic period" : 42 }

The Scala standard library

Upwardly until recently I did not know that Scala had a JSON utility in its standard library. Just here it is!

Parsing raw JSON

scala > import scala . util . parsing . json . _ import scala . util . parsing . json . _ scala > val rawJson = "" "{" howdy ": " globe ", " age ": 42}" "" rawJson : String = { "howdy" : "world" , "age" : 42 } scala > JSON . parseFull ( rawJson ) alert : in that location was ane deprecation warning ; re-run with -deprecation for details res0 : Pick [ Whatsoever ] = Some ( Map ( hi -> world , age -> 42.0 ) )

Browsing the "AST"

scala > res0 . get . asInstanceOf [ Map [ String , Whatever ] ] ( "hullo" ) . asInstanceOf [ String ] res4 : Cord = world

Even more!

I am forgetting a ton of libraries here I am sure. That, and I am tired and my glass of Uigeadail is getting empty.

Then let me mention a few more:

• Julien Richard-Foy built the play-json-variants which add the root hierarchy capability to play-json
• Pascal Voitot added a few functional manipulations to play-json in play-json-attachment
• Robert J. Macomber congenital the rojoma-json library which I only discovered now and am as well tired to cover (pitiful)

Great, at present which one to selection?

Honestly I don't have whatever good communication hither. These days I am sticking to play-json and sphere-json, but that's for no item reason other than these libraries being there and doing what I need to do (parse and write JSON, traverse and some rare times transform the AST, bind to case classes, make it possible to support custom Java types). If play-json had support for hierarchies out of the box I would probably not have even looked for anything else.

Because for all the joy in that location seems to be in implementing JSON libraries in Scala, one thing has to be said: JSON de/serialization is wearisome. It's this annoying matter that you have to do in social club to get your application to talk to another computerized system, period.

I take never met a developer who told me how much pleasance they derived from turning classes into JSON strings and back. That'due south simply not a thing.

I have, yet, met more one developer that has run into problem getting library X to encompass i of the uncomplicated utilise-cases outlined higher up. Believe me, there is nothing more frustrating than having to spend fourth dimension on the annoying chore of setting upward JSON de/serialization in order to practice the boring thing of tossing strings dorsum and forth the network. That'due south time you will never get back.

Adept dark, and proficient luck.

harrisonansters.blogspot.com

Source: https://manuel.bernhardt.io/2015/11/06/a-quick-tour-of-json-libraries-in-scala/

Share this post