Parsing JSON and creating analytics out of the data, what is the best way to do it?

Question

I am on a project where I call an API and I want to make statistics with the data returned. It returns a big .json object. As that is not possible to flatten, and I am not interested on all the data returned either, I want to parse only certain keys. I have thought of parsing and then creating a well structured json myself, and from there work with it with panda (I am coding this with Python)

Would this be a good approach? Is it even necessary to create a new .json structure to hold the parsed data?

• dagnelies: that approach seems perfectly fine to me
• xavier: I think I will go for dictionaries, as I can fill them with the variables I need and I don't need to create a .json file. My main concern is to avoid creating any file or database. I just want to query the API database and show the statistics without creating a database or a file.

Answer 1

what i would do is look for the interesting data and make tables and colums in a database accordingly, then parse the data the the database. you can then leverage the power of SQL the get your analytics data

• xavier: I was thinking that i do not want to create another database, as i am getting The data out of queries made to the system that provides the API, that already has a database.
• xavier: Maybe hold the data of the query on a dictionary object would be a better choice ?
• Bert-jan: in that case take a look at LINQ, its for c# but there must be python implementations somewhere. LINQ has the ability to query arrays and dictionary's much like SQL. code.msdn.microsoft.com/101-LINQ-Samples-3fb9811b
• Adam Zuckerman: This doesn't answer the question.