We want to flatten above structure using explode API of data frames. Whatever samples that we got from the documentation and git is talking about exploding a String by splitting but here we have an Array strucutre. We did not get any examples for this in web also. Or I could be missing something..

Could anyone please help in using explode method with nested array strucutre..

Re: Explode function in Data Frames

This is Chris Fregly from Databricks. I just talked to my co-worker, Michael Armbrust (Spark SQL, Catalyst, DataFrame guru), and we came up with the code sample below. Hopefully, this is what you're looking for.

Michael admits that this is a bit verbose, so he may implement a more condense `explodeArray()` method on DataFrame at some point.

Re: Explode function in Data Frames

Thanks for your quick response and solution.. Let me try this solution in our data model.. I will update this thread about the outcome.. Many thanks again.. explodeArray would be great option.. Nevetheless this solution is really cool.. Thanks again for you time..

Re: Explode function in Data Frames

This info is very helpful, but I've got a twist that I can't seem to figure out.

I got this working for a single level of depth, but I'm somehow challenged (and a Scala noob) by multiple levels. Below is the schema of my DataFrame (modelled after the HL7 FHIR DSTU2 specification, read from a parquet file). I am trying to explode out the individual values in the "given" field of the "name" struct array (so, a nested array), for example, but following the initial explode of the name array, the field I exploded to (called "nar") is not an array of struct, it's simply an array of String, which I think is challenging to the explode() method. I've tried a number of different approaches, but haven't found the right combination. I need to be able to test each individual "given" names (and other values in the struct) against those values in other records (name matching). Do you think this is possible?