Parse data into struct efficiently by reducing memory footprint

Question

I have a simple program where I am deserializing my bytes byteSlice into a ClientProduct array struct. And then I iterate over that ClientProduct array and construct definitions.CustomerProduct struct for each ClientProduct.

Below is my struct details:

type ProductInfo struct {
 Name map[Locale]map[int]string
}
type Locale string
type CustomerProduct struct {
 Prs string
 ProductId int64
 Catalogs []int32
 Categories []int
 BaseProductId *int64
 StatusCode int
 IsActive bool
 Common
}
type ValueMetrics struct {
 Value string
 ClientId int64
}
type Common struct {
 Leagues []ValueMetrics
 Teams []ValueMetrics
 ProductInfo
}

Here is the code which deserializes byteSlice into ClientProduct array struct by using Convert and few other helper methods.

var productRows []ClientProduct
err = json.Unmarshal(byteSlice, &productRows)
if err != nil {
 return errs.Wrap(err)
}
for i := range productRows {
 var flatProduct definitions.CustomerProduct
 err = r.Convert(spn, &productRows[i], &flatProduct)
 if err != nil {
 return errs.Wrap(err)
 }
 if flatProduct.StatusCode == definitions.DONE {
 continue
 }
 // populate map here from each "flatProduct"
}

Here is my Convert method which populates my CustomerProduct struct. And few other helper methods which help to populate CustomerProduct struct

func (r *clientRepo) Convert(span log.Span, in *ClientProduct, out *definitions.CustomerProduct) error {
 commons, err := r.getCommonStruct(span, in)
 if err != nil {
 return errs.Wrap(err)
 }
 out.Common = commons
 out.Catalogs = r.convertParquetIntSliceToInt32Slice(in.Catalogs)
 out.Categories = in.Categories
 out.ProductId = in.Id
 out.IsActive = in.IsActive
 out.Prs = in.PRS
 out.StatusCode = in.StatusCode
 return nil
}
func (r *clientRepo) getCommonStruct(span log.Span, in *ClientProduct) (definitions.Common, error) {
 out := definitions.Common{}
 for _, pv := range in.PropertyValues {
 switch pv.Name {
 case "ABC":
 out.Leagues = r.concatStringSlice(pv, out.Leagues)
 case "DEF":
 out.Teams = r.concatStringSlice(pv, out.Teams)
 }
 }
 for _, opv := range in.OverriddenPropertyValues {
 switch opv.Name {
 case "CustomerName":
 if opv.Locale == "" {
 // log error
 } else {
 out.Name = r.attach(definitions.Locale(opv.Locale), opv, out.Name)
 }
 }
 }
 return out, nil
}
func (r *clientRepo) concatStringSlice(pv PropertyValue, list []definitions.ValueMetrics) []definitions.ValueMetrics {
 treeValue := definitions.ValueMetrics{Value: pv.Value, ClientId: pv.ClientId}
 if list == nil {
 return []definitions.ValueMetrics{treeValue}
 }
 return append(list, treeValue)
}
func (r *clientRepo) attach(locale definitions.Locale, opv OverriddenPropertyValue, m map[definitions.Locale]map[int]string) map[definitions.Locale]map[int]string {
 if m == nil {
 om := map[int]string{
 opv.GroupId: opv.Value,
 }
 return map[definitions.Locale]map[int]string{
 locale: om,
 }
 }
 os, ok := m[locale]
 if ok {
 os[opv.GroupId] = opv.Value
 m[locale] = os
 } else {
 m[locale] = map[int]string{
 opv.GroupId: opv.Value,
 }
 }
 return m
}
func (r *clientRepo) convertParquetIntSliceToInt32Slice(catalogs []int) []int32 {
 var out []int32
 for _, catalog := range catalogs {
 out = append(out, int32(catalog))
 }
 return out
}

Problem Statement

Above code works fine. Basically looking for code review on the way I am populating "CustomerProduct" using all these helper methods. Anything that can be improved?

• I am looking for inputs or ideas where I can improve the way I am populating CustomerProduct struct by using all those helper methods I have starting with Convert method.
• Also is there anything that can be improved in above code which can help in our memory footprint? Like using pointer's or passing reference of it instead of direct passing.

I am seeing some memory fluctuating a lot whenever this deserialization happens and CustomerProduct struct is populated from all those helper methods. Any guidance will be greatly appreciated.

Answer 1

It's important to have a benchmark before you start trying to optimize the code. That being said, I can think of a few general recommendations.

• The ProductInfo.Name field is a nested map, this will require a lot of maps to be allocated. Instead of nested maps, consider defining a struct key type, something like

  type NameKey struct {
   Locale Locale
   GroupID int
  }
  type ProductInfo struct {
   Name map[NameKey]string
  }
  

  Then getCommonStruct might look like

  func (r *clientRepo) getCommonStruct(span log.Span, in ClientProduct) (definitions.Common, error) {
   out := definitions.Common{
   // Initialize Name ahead of time.
   ProductInfo: definitions.ProductInfo {
   Name: make(map[definitions.NameKey]string),
   },
   }
   for _, pv := range in.PropertyValues {
   // This part is the same.
   ...
   }
   for _, opv := range in.OverriddenPropertyValues {
   switch opv.Name {
   case "CustomerName":
   if opv.Locale == "" {
   ...
   continue
   }
   // Just set the value in the map.
   out.Name[NameKey{Locale: definitions.Locale(opv.Locale), GroupID: opv.GroupID}] = opv.Value
   }
   }
   return out, nil
  }
  

• In general prefer passing values instead of pointers. This tends to produce code that is easier to read and reason about. Computers are really fast at copying memory :) Of course, if your benchmark tells you a pointer is faster somewhere, feel free to use that.

• The Convert method can return a CustomerProduct instead of using an out parameter. I would also recommend taking the ClientProduct by value since it isn't modified.

• The concatStringSlice helper can be simplified, append works just fine on nil:

  func (r *clientRepo) concatStringSlice(pv PropertyValue, list []definitions.ValueMetrics) []definitions.ValueMetrics {
   return append(list, definitions.ValueMetrics{Value: pv.Value, ClientId: pv.ClientId})
  }
  

• In convertParquetIntSliceToInt32Slice, out can be pre-allocated to the right size:

  func (r *clientRepo) convertParquetIntSliceToInt32Slice(catalogs []int) []int32 {
   out := make([]int32, 0, len(catalogs))
   for _, catalog := range catalogs {
   out = append(out, int32(catalog))
   }
   return out
  }
  

Sorry for the typos it's late :)

• AndyP: Thanks for your good suggestion. I am looking into all those. If possible can you add example on pt1 and pt4 on how those method will look like with suggested change? This will help me understand better.
• rose: Updated with some examples of how the methods might look
• AndyP: Thanks a lot. Appreciate it. In the attach method, I was mainly looking for if m is nil then how should I deal with that. Any thoughts?
• rose: Set out.Name = make(map[NameKey]string) before calling attach. It simplifies the attach function to assume the map has already been initialized.
• AndyP: I see. So my attach function returns the map but in your case it doesn't return it right? so my else block will look like this then? else { out.Name = make(map[NameKey]string) out.Name = r.attach(definitions.Locale(opv.Locale), opv, out.Name) } or I got it wrong? I initialize the map before calling attach. Is that what you meant?