Processing a company list in which every element is an employee list

Question

Now, there is a company list in which an element is an employee list in every company. I calculate the salary of employees in every company with a RESTful API one by one,. This is very slow, so I join 1000 elements to one list and post it to the service.

import requests
def _calculate_employee_salaries(employees):
 response = requests.post(url, json={"input": employees})
 salaries = parse_response(response)
 return salaries
def get_calculate_employee_salaries(company_employees_lis):
 begin, step = 0, 1000
 company_employee_salaries_lis = []
 while begin < len(company_employees_lis):
 employees = []
 employee_num_lis = []
 for comp_employees in company_employees_lis[begin:(begin+step)]:
 employees.extend(comp_employees)
 employee_num_lis.append(len(comp_employees))
 salaries = _calculate_employee_salaries(employees)
 idx = 0
 for num in employee_num_lis:
 company_employee_salaries_lis.append(salaries[idx:(idx + num)])
 idx += num
 begin += step
 return company_employee_salaries_lis
if __name__ == "__main__":
 company_employees_lis = [["employees_id_1", "employees_id_2", "employees_id_3"], ["employees_id_4"], ["employees_id_5", "employees_id_6"]]
 company_employee_salaries_lis = get_calculate_employee_salaries(company_employees_lis)

I think my code is bad, because I use a variable employee_num_lis to remember the number of employees in every company. And there must be clean, artful and pythonic code. This code is just a sample, not what I should do.

Answer 1

Yes, you can simplify this. Let's break down the algorithm needed for this:

1. Chain all employees into one stream
2. Chunk this stream into 1000 employees
   • For each chunk get the salaries
3. Distribute the salaries back again into the different companies

The first part can be done using itertools.chain.from_iterable. The second one can use e.g. this grouper recipe. And the final part can be easily done with itertools.islice if we keep everything before as generators.

from itertools import chain, islice

def grouper(iterable, n):
 it = iter(iterable)
 return iter(lambda: tuple(islice(it, n)), ())

def get_salaries(employees):
 respone = requests.post(url, json={"input": employees})
 return parse_respone(respone)
def employee_salaries_per_company(company_employees):
 all_employees = chain.from_iterable(company_employees) 
 salaries = chain.from_iterable(get_salaries(employees)
 for employees in grouper(all_employees, 1000))
 return [list(islice(salaries, n)) for n in map(len, company_employees)]

---

Note that in english the plural of employee is employees and not emploices; and the plural of salary is salaries and not salarices.