How to avoid nested map allocations in below code?

Question

I have a below struct where I have a nested map for CustomersIndex which allocates bunch of internal maps causing memory increase. I profiled it so I noticed this. I am trying to see if there is any way to redesign my CustomersIndex data structure which doesn't uses nested map?

const (
 departmentsKey = "departments"
)
type CustomerManifest struct {
 Customers []definitions.Customer
 CustomersIndex map[int]map[int]definitions.Customer
}

This is the way it is being populated here in my below code:

func updateData(mdmCache *mdm.Cache) map[string]interface{} {
 memCache := mdmCache.MemCache()
 var customers []definitions.Customer
 var customersIndex = map[int]map[int]definitions.Customer{}
 for _, r := range memCache.Customer {
 customer := definitions.Customer{
 Id: int(r.Id),
 SetId: int(r.DepartmentSetId),
 }
 customers = append(customers, customer)
 _, yes := customersIndex[customer.SetId]
 if !yes {
 customersIndex[customer.SetId] = make(map[int]definitions.Customer)
 }
 customersIndex[customer.SetId][customer.Id] = customer
 }
 return map[string]interface{}{
 departmentsKey: &CustomerManifest{Customers: customers, CustomersIndex: customersIndex},
 }
}

And this is the way I am getting my CustomersIndex nested map.

func (c *Client) GetCustomerIndex() map[int]map[int]definitions.Customer {
 c.mutex.RLock()
 defer c.mutex.RUnlock()
 customersIndex := c.data[departmentsKey].(*CustomerManifest).CustomersIndex
 return customersIndex
}

Is there any way to design my CustomersIndex in a way where I don't have to use nested map? Also can nested map cause more allocations causing more memory increase?

• pacmaninbw: Please change the title to indicate what the code is used for. That will help us review the code. Please read How do I ask a good question?.

Answer 1

I presume the definitions.Customer struct has more in it than just the Id and SetId fields? For example, a Name or other fields. (Otherwise the index wouldn't be useful.)

It may help to use a single map with a composite key. For example, define a key type with two int fields and index it by customer ID and department ("set ID"):

type indexKey struct {
 id int
 setId int
}
var CustomersIndex map[indexKey]Customer
// ...
fmt.Println(CustomersIndex[indexKey{id: 1, setId: 2}])

Full example in Go Playground.

I suspect this will save memory to use a single map, but I'm not 100% sure. You'd need to compare the profiling results for your data.

Is it actually using too much memory the original way, or is this a theoretical concern?