Retrieve, parse and export data to tables via sqlalchemy

Question 1

The project I've working on is designed with the intent of being able to quickly and efficiently identify discrepancies in live sports betting data that comes from web API's in json format. The first step is to retrieve the json's and I have incorporated async for that. Each json is structured differently so it gets parsed within it's own class called GamesDict. Once parsed it gets inserted to a database table via sqlalchemy and is ready for evaluation. A key point here is that the json files contain basic data for every game, but we also need more data that can only be obtained via a separate json URL. URL's for individual games need event_id's so that is obtained within the db_query script. Once the individual game json's for each source are obtained they can be parsed and inserted to tables similar to how it was done in the first process.

This is the first iteration of the project and I know I have a lot to learn and would appreciate any feedback.

These are the issues I'm dealing with right now:

Stick with core or use ORM?
How to organize the data - more tables or less?
How to move/replace outdated entries with most recent

parse_games.py

BOOK = 'book3'
class GamesDict:
 """
 Class GamesDict takes a list of dictionaries and a database control object as variables.
 It calls on methods to parse the data pursuant to it's stat type (runline, props, etc)
 and then commits the data to it's corresponding table.
 
 """
 
 def __init__(self, listofdics, db_control):
 self._dic = listofdics[BOOK]['eventGroup']
 self.db_control = db_control
 
 self.events = self._dic.get('events',())
 self.categories = self._dic.get('offerCategories',())
 self.export_gamelist()
 self.export_runline()
 self.export_moneyline()
 self.export_total()
 self.export_propshomeruns()
 self.export_propshits()
 self.export_propsrbis()
 self.export_propsrunsscored()
 self.export_propsstolenbases()
 self.export_propssingles()
 self.export_propsdoubles()
 self.export_propstriples()
 self.export_propsstrikeouts()
 self.export_propsoutsrecorded()
 self.export_propswin()
 self.export_propshitsallowed()
 self.export_propswalks()
 self.export_propsearnedruns()
## self.export_plateappearanceexact()
 def export_gamelist(self):
 self.gamelist = Gamelist().to_dict(self.events)
 GamelistControl(BOOK,self.db_control).commit_entry(self.gamelist)
 
 def export_runline(self):
 self.runline = Runline().to_dict(self.categories)
 RunlineControl(BOOK,self.db_control).commit_entry(self.runline)
 def export_moneyline(self):
 self.moneyline = Moneyline().to_dict(self.categories)
 MoneylineControl(BOOK,self.db_control).commit_entry(self.moneyline)
 
 def export_total(self):
 self.total = Total().to_dict(self.categories)
 TotalControl(BOOK,self.db_control).commit_entry(self.total)
 def export_propshomeruns(self):
 self.propshomeruns = PropsHomeruns().to_dict(self.categories)
 PropsControl(BOOK,self.db_control,'homeruns').commit_entry(self.propshomeruns)
 def export_propshits(self):
 self.propshits = PropsHits().to_dict(self.categories)
 PropsControl(BOOK,self.db_control,'hits').commit_entry(self.propshits)
 def export_propstotalbases(self):
 self.propstotalbases = PropsTotalBases().to_dict(self.categories)
 PropsControl(BOOK,self.db_control,'totalbases').commit_entry(self.propstotalbases)
 def export_propsrbis(self):
 self.propsrbis = PropsRBIs().to_dict(self.categories)
 PropsControl(BOOK,self.db_control,'rbis').commit_entry(self.propsrbis)
 def export_propsrunsscored(self):
 self.propsrunsscored = PropsRunsScored().to_dict(self.categories)
 PropsControl(BOOK,self.db_control,'runsscored').commit_entry(self.propsrunsscored)
 def export_propsstolenbases(self):
 self.propsstolenbases = PropsStolenBases().to_dict(self.categories)
 PropsControl(BOOK,self.db_control,'stolenbases').commit_entry(self.propsstolenbases)
 def export_propssingles(self):
 self.propssingles = PropsSingles().to_dict(self.categories)
 PropsControl(BOOK,self.db_control,'singles').commit_entry(self.propssingles)
 def export_propsdoubles(self):
 self.propsdoubles = PropsDoubles().to_dict(self.categories)
 PropsControl(BOOK,self.db_control,'doubles').commit_entry(self.propsdoubles)
 def export_propstriples(self):
 self.propstriples = PropsTriples().to_dict(self.categories)
 PropsControl(BOOK,self.db_control,'triples').commit_entry(self.propstriples)
 def export_propsstrikeouts(self):
 self.propsstrikeouts = PropsStrikeouts().to_dict(self.categories)
 PropsControl(BOOK,self.db_control,'strikeouts').commit_entry(self.propsstrikeouts)
 def export_propsoutsrecorded(self):
 self.propsoutsrecorded = PropsOutsRecorded().to_dict(self.categories)
 PropsControl(BOOK,self.db_control,'outsrecorded').commit_entry(self.propsoutsrecorded)
 def export_propswin(self):
 self.propswin = PropsWin().to_dict(self.categories)
 PropsControl(BOOK,self.db_control,'win').commit_entry(self.propswin)
 def export_propshitsallowed(self):
 self.propshitsallowed = PropsHitsAllowed().to_dict(self.categories)
 PropsControl(BOOK,self.db_control,'hitsallowed').commit_entry(self.propshitsallowed)
 def export_propswalks(self):
 self.propswalks = PropsWalks().to_dict(self.categories)
 PropsControl(BOOK,self.db_control,'walks').commit_entry(self.propswalks)
 def export_propsearnedruns(self):
 self.propsearnedruns = PropsEarnedRuns().to_dict(self.categories)
 PropsControl(BOOK,self.db_control,'earnedruns').commit_entry(self.propsearnedruns)
##
## def export_plateappearanceexact(self):
## self.plateappearanceexact = PAExact().to_dict(self.categories)
## PAExact(BOOK,self.db_control).commit_entry(self.plateappearanceexact)
 
 
class Parser:
 CATEGORY: int
 SUBCATEGORY: int
 DISPLAY_NAME: str
 def to_dict(self, markets):
 return [market for market in self._get_records(markets)]
 @classmethod
 def _get_records(cls, data):
 raise NotImplementedError()
class Gamelist(Parser):
 """
 Class Gamelist is a dedicated parser for general game information
 such as event id and start time. 
 
 """
 
 CATEGORY = None
 SUBCATEGORY = None
 DISPLAY_NAME = 'Gamelist'
 @classmethod
 def _get_records(cls, events)
 for event in events:
 if "@" in event['name']:
 yield from cls._get_data(event)
 
 @classmethod
 def _get_data(cls, event)
 event_id = event['eventId']
 game_datetime = unix(event['startDate'])
 away_abbr = TEAMS.get(event["teamName1"])
 home_abbr = TEAMS.get(event["teamName2"])
 game_ref = (f'{away_abbr}@{home_abbr}')
 return ((event_id, game_ref, game_datetime, BOOK)),
 
class Category(Parser):
 """
 Class Category takes the larger dictionary and parses it based on the
 statistics specific category and subcategory. 
 
 """
 @classmethod
 def _get_records(cls, data)
 for item in data:
 if item['offerCategoryId'] == cls.CATEGORY:
 yield from cls._get_subcatdes(item['offerSubcategoryDescriptors'])
 @classmethod
 def _get_subcatdes(cls, data)
 for item in data:
 if item['subcategoryId'] == cls.SUBCATEGORY:
 yield from cls._get_subcat(item['offerSubcategory'].get('offers'))
 @classmethod
 def _get_subcat(cls, data)
 for item in data:
 for x in item:
 if cls.DISPLAY_NAME in x.get('label',()) and x.get('isOpen',()):
 yield from cls._get_data(x)
class Runline(Category):
 """
 Class Runline is a dedicated parser for runline statistics. 
 """
 
 CATEGORY = 493
 SUBCATEGORY = 4519
 DISPLAY_NAME = 'Run Line'
 
 @classmethod
 def _get_data(cls, market):
 event_id = market['eventId']
 away_team, home_team = market['outcomes'][:2]
 away_abbr = TEAMS.get(away_team['label'])
 home_abbr = TEAMS.get(home_team['label'])
 away_odds = away_team['oddsDecimal']
 home_odds = home_team['oddsDecimal']
 away_line = away_team['line']
 home_line = home_team['line']
 return (
 (away_abbr, event_id, away_odds, away_line, BOOK),
 (home_abbr, event_id, home_odds, home_line, BOOK),
 )
class Moneyline(Category):
 """
 Class Moneyline is a dedicated parser for moneyline statistics. 
 """
 
 CATEGORY = 493
 SUBCATEGORY = 4519
 DISPLAY_NAME = 'Moneyline'
 
 @classmethod
 def _get_data(cls, market):
 event_id = market['eventId']
 away_team, home_team = market['outcomes'][:2]
 away_abbr = TEAMS.get(away_team['label'])
 home_abbr = TEAMS.get(home_team['label'])
 away_odds = away_team['oddsDecimal']
 home_odds = home_team['oddsDecimal']
 return (
 (away_abbr, event_id, away_odds, BOOK),
 (home_abbr, event_id, home_odds, BOOK),
 )
class Total(Category):
 """
 Class Total is a dedicated parser for total statistics. 
 """
 
 CATEGORY = 493
 SUBCATEGORY = 4519
 DISPLAY_NAME = 'Total'
 
 @classmethod
 def _get_data(cls, market):
 event_id = market['eventId']
 
 away_team, home_team = market['outcomes'][:2]
 line = away_team['line']
 over = away_team['oddsDecimal']
 under = home_team['oddsDecimal']
 return (
 (event_id, line, over, under, BOOK),
 )
class Props(Category):
 """
 Class Props is a dedicated parser for all props statistics.
 """
 
 @classmethod
 def _get_data(cls, market):
 event_id = market['eventId']
 cont1, cont2 = market['outcomes'][:2]
 player = cont1.get('participant','')
 line = cont1.get('line',0)
 over = cont1.get('oddsDecimal','')
 under = cont2.get('oddsDecimal','')
 return (
 (player, event_id, line, over, under, BOOK),
 )
class PropsHomeruns(Props):
 CATEGORY = 743
 SUBCATEGORY = 6606
 DISPLAY_NAME = 'Home Runs'
class PropsHits(Props):
 CATEGORY = 743
 SUBCATEGORY = 6719
 DISPLAY_NAME = 'Hits'
 
class PropsTotalBases(Props):
 CATEGORY = 743
 SUBCATEGORY = 6606
 DISPLAY_NAME = 'Total Bases'
class PropsRBIs(Props):
 CATEGORY = 743
 SUBCATEGORY = 8025
 DISPLAY_NAME = 'RBIs'
class PropsRunsScored(Props):
 CATEGORY = 743
 SUBCATEGORY = 7979
 DISPLAY_NAME = 'Runs'
class PropsStolenBases(Props):
 CATEGORY = 743
 SUBCATEGORY = 9872
 DISPLAY_NAME = 'Stolen Bases'
class PropsSingles(Props):
 CATEGORY = 743
 SUBCATEGORY = 11031
 DISPLAY_NAME = 'Singles'
class PropsDoubles(Props):
 CATEGORY = 743
 SUBCATEGORY = 11032
 DISPLAY_NAME = 'Doubles'
class PropsTriples(Props):
 CATEGORY = 743
 SUBCATEGORY = 11033
 DISPLAY_NAME = 'Triples'
class PropsStrikeouts(Props):
 CATEGORY = 1031
 SUBCATEGORY = 9885
 DISPLAY_NAME = 'Strikeouts'
class PropsOutsRecorded(Props):
 CATEGORY = 1031
 SUBCATEGORY = 9883
 DISPLAY_NAME = 'Outs'
class PropsWin(Props):
 CATEGORY = 1031
 SUBCATEGORY = 9884
 DISPLAY_NAME = 'Win?'
class PropsHitsAllowed(Props):
 CATEGORY = 1031
 SUBCATEGORY = 9886
 DISPLAY_NAME = 'Hits Allowed'
class PropsWalks(Props):
 CATEGORY = 1031
 SUBCATEGORY = 11035
 DISPLAY_NAME = 'Walks'
class PropsEarnedRuns(Props):
 CATEGORY = 1031
 SUBCATEGORY = 11064
 DISPLAY_NAME = 'Earned Runs'

db_control.py

class DBControl:
 def __init__(self):
 logging.info('[DBControl]: CONNECTING TO DATABASE')
 self.engine = create_engine('sqlite:///games.db', echo=False)
 self.inspector = inspect(self.engine)
 self.db_connection = self.engine.connect()
 self.create_session = sessionmaker(bind=self.engine)
 self.metadata = MetaData(bind=self.engine)
class GamelistMLBControl:
 def __init__(self,book,db_control):
 self.table_name = f'{book}_gamelist'
 self.db_control = db_control
 
 self.table = self.check_table()
 
 def commit_entry(self,site_data):
 write_session = scoped_session(self.db_control.create_session)
 insert_stmt = insert(self.table).values(site_data)
 write_session.execute(insert_stmt)
 write_session.commit()
 write_session.remove()
 def check_table(self):
 metadata = self.db_control.metadata
 table_name = self.table_name
 if table_name not in self.db_control.inspector.get_table_names():
 table_name = Table(
 str(table_name),
 metadata,
 Column("event_id", Integer, primary_key=True),
 Column("game_ref", String),
 Column("game_datetime", Integer),
 Column("book", String),
 )
 
 metadata.create_all(self.db_control.engine)
 logging.info(f'[DBControl]: {table_name} created')
 else:
 metadata.reflect(self.db_control.engine)
 logging.info(f'[DBControl]: {table_name} exists')
 return Table(table_name, metadata, autoload=True)
 
class GamelistControl:
 def __init__(self,book,db_control):
 self.table_name = f'{book}_gamelist'
 self.db_control = db_control
 self.table = self.check_table()
 
 def commit_entry(self,site_data):
 write_session = scoped_session(self.db_control.create_session)
 insert_stmt = insert(self.table).values(site_data)
 write_session.execute(insert_stmt)
 write_session.commit()
 write_session.remove()
 def check_table(self):
 metadata = self.db_control.metadata
 table_name = self.table_name
 if table_name not in self.db_control.inspector.get_table_names():
 table_name = Table(
 str(table_name),
 metadata,
 Column("event_id", Integer, primary_key=True),
 Column("game_ref", String, primary_key=True),
 Column("game_datetime", Integer, primary_key=True),
 Column("book", String),
 ForeignKeyConstraint(['game_ref', 'game_datetime'],
 ['mlb_gamelist.game_ref', 'mlb_gamelist.game_datetime'])
 )
 
 metadata.create_all(self.db_control.engine)
 logging.info(f'[DBControl]: {table_name} created')
 else:
 metadata.reflect(self.db_control.engine)
 logging.info(f'[DBControl]: {table_name} exists')
 return Table(table_name, metadata, autoload=True)
 
class RunlineControl:
 def __init__(self,book,db_control):
 self.table_name = f'{book}_runline'
 self.db_control = db_control
 def commit_entry(self,site_data):
 write_session = scoped_session(self.db_control.create_session)
 insert_stmt = insert(self.check_table()).values(site_data)
 write_session.execute(insert_stmt)
 write_session.commit()
 write_session.remove()
 def check_table(self):
 metadata = MetaData(bind=self.db_control.engine)
 if self.table_name not in self.db_control.inspector.get_table_names():
 table_name = Table(
 str(self.table_name),
 metadata,
 Column("team", String, primary_key=True),
 Column("event_id", Integer, primary_key=True),
 Column("odds", Integer),
 Column("line", Float),
 Column("book", String, primary_key=True),
 PrimaryKeyConstraint("event_id", "team", "book"),
 )
 
 metadata.create_all(self.db_control.db_connection)
 logging.info(f'[DBControl]: {table_name} created')
 else:
 metadata.reflect(self.db_control.engine)
 logging.info(f'[DBControl]: {table_name} exists')
 return Table(table_name, metadata, autoload=True)
class MoneylineControl:
 def __init__(self,book,db_control):
 self.table_name = f'{book}_moneyline'
 self.db_control = db_control
 def commit_entry(self,site_data):
 write_session = scoped_session(self.db_control.create_session)
 insert_stmt = insert(self.check_table()).values(site_data)
 write_session.execute(insert_stmt)
 write_session.commit()
 write_session.remove()
 def check_table(self):
 metadata = MetaData(bind=self.db_control.engine)
 if self.table_name not in self.db_control.inspector.get_table_names():
 table_name = Table(
 str(self.table_name),
 metadata,
 Column("team", String, primary_key=True),
 Column("event_id", Integer, primary_key=True),
 Column("odds", Integer),
 Column("book", String, primary_key=True),
 PrimaryKeyConstraint("event_id", "team", "book"),
 )
 
 metadata.create_all(self.db_control.db_connection)
 logging.info(f'[DBControl]: {table_name} created')
 else:
 metadata.reflect(self.db_control.engine)
 logging.info(f'[DBControl]: {table_name} exists')
 return Table(table_name, metadata, autoload=True)
class TotalControl:
 def __init__(self,book,db_control):
 self.table_name = f'{book}_total'
 self.db_control = db_control
 def commit_entry(self,site_data):
 write_session = scoped_session(self.db_control.create_session)
 insert_stmt = insert(self.check_table()).values(site_data)
 write_session.execute(insert_stmt)
 write_session.commit()
 write_session.remove()
 def check_table(self):
 metadata = MetaData(bind=self.db_control.engine)
 if self.table_name not in self.db_control.inspector.get_table_names():
 table_name = Table(
 str(self.table_name),
 metadata,
 Column("event_id", Integer, primary_key=True),
 Column("line", Float),
 Column("over", Integer),
 Column("under", Integer),
 Column("book", String, primary_key=True),
 PrimaryKeyConstraint("event_id", "book"),
 )
 
 metadata.create_all(self.db_control.db_connection)
 logging.info(f'[DBControl]: {table_name} created')
 else:
 metadata.reflect(self.db_control.engine)
 logging.info(f'[DBControl]: {table_name} exists')
 return Table(table_name, metadata, autoload=True)
class PropsControl:
 def __init__(self,book,db_control,prop):
 self.table_name = f'{book}_props{prop}'
 self.db_control = db_control
 def commit_entry(self,data):
 write_session = scoped_session(self.db_control.create_session)
 insert_stmt = insert(self.check_table()).values(data)
 write_session.execute(insert_stmt)
 write_session.commit()
 write_session.remove()
 def check_table(self):
 metadata = MetaData(bind=self.db_control.engine)
 if self.table_name not in self.db_control.inspector.get_table_names():
 table_name = Table(
 str(self.table_name),
 metadata,
 Column("player", String, primary_key=True),
 Column("event_id", Integer, primary_key=True),
 Column("line", Float),
 Column("over", Integer),
 Column("under", Integer),
 Column("book", String, primary_key=True),
 PrimaryKeyConstraint("event_id", "player", "book"),
 )
 
 metadata.create_all(self.db_control.db_connection)
 logging.info(f'[DBControl]: {table_name} created')
 else:
 metadata.reflect(self.db_control.engine)
 logging.info(f'[DBControl]: {table_name} exists')
 return Table(table_name, metadata, autoload=True)

db_query.py

from sqlalchemy import (Table, MetaData)
class DBQuery:
 def __init__(self,books,db_control):
 self.books = books
 self.db_control = db_control
 self.session = self.db_control.create_session
 
 def gamelist(self,caesars_ids):
 meta = MetaData(self.db_control.engine)
 t1 = Table('mlb_gamelist', meta, autoload=True)
 t2 = Table('caesars_gamelist', meta, autoload=True)
 t3 = Table('fanduel_gamelist', meta, autoload=True)
 t4 = Table('draftkings_gamelist', meta, autoload=True)
 t5 = Table('pointsbet_gamelist', meta, autoload=True)
 t6 = Table('mgm_gamelist', meta, autoload=True)
 sess = self.session()
 data = sess.query(
 t1.c.event_id,
 t2.c.event_id,
 t3.c.event_id,
 t4.c.event_id,
 t5.c.event_id,
 t6.c.event_id
 ).join(
 t2
 ).join(
 t3
 ).join(
 t4
 ).join(
 t5
 ).join(
 t6
 )
 lst = []
 for x in data:
 dic = {'caesars': caesars_ids.get(x[1]),
 'fanduel': x[2],
 'draftkings': x[3],
 'pointsbet': x[4],
 'mgm': x[5]
 }
 lst.append(dic)
 return lst

app.py

import logging
from time import sleep
from pprint import pprint
from config import configs
from extract.download import get_games, get_game
from db.db_control import DBControl
from db.db_query import DBQuery
from pymlb.parse_games import GamesDict as MLBGamesDict
from pycaesars.parse_games import GamesDict as CaesarsGamesDict
from pycaesars.parse_game import GameDict as CaesarsGameDict
from pyfanduel.parse_games import GamesDict as FanduelGamesDict
from pyfanduel.parse_game import GameDict as FanduelGameDict
from pydraftkings.parse_games import GamesDict as DraftkingsGamesDict
from pypointsbet.parse_games import GamesDict as PointsbetGamesDict
from pypointsbet.parse_game import GameDict as PointsbetGameDict
from pymgm.parse_games import GamesDict as MGMGamesDict
from pymgm.parse_game import GameDict as MGMGameDict
from utils import delete_file
if __name__ == "__main__":
 logging.basicConfig(level=logging.DEBUG)
 delete_file('games.db')
 booklist = [k for k,v in configs['books'].items() if v]
 logging.warning(f'[APP]: Retrieving GAMES data for {booklist}')
 games = get_games()
 db_control = DBControl()
 if configs['books']['mlb']:
 MLBGamesDict(games,db_control)
 if configs['books']['caesars']:
 caesars = CaesarsGamesDict(games,db_control)
 caesars_ids = caesars.id_dict
 if configs['books']['fanduel']:
 FanduelGamesDict(games,db_control)
 if configs['books']['draftkings']:
 DraftkingsGamesDict(games,db_control)
 if configs['books']['pointsbet']:
 PointsbetGamesDict(games,db_control)
 if configs['books']['mgm']:
 MGMGamesDict(games,db_control)
 logging.info(f'[APP]: Retrieving GAME data')
 game = DBQuery(booklist,db_control).gamelist(caesars_ids)
 data = []
 for event_ids in game:
 games = get_game(event_ids)
 data.append(games)
 sleep(1)
 if configs['books']['caesars']:
 CaesarsGameDict(data,db_control)
 if configs['books']['fanduel']:
 FanduelGameDict(data,db_control)
 if configs['books']['pointsbet']:
 PointsbetGameDict(data,db_control)
 if configs['books']['mgm']:
 MGMGameDict(data,db_control)

parse_games.py REVISED

class Parser:
 DISPLAY_NAME: str
 def __init__(self, dic):
 self._dic = dic[BOOK]['eventGroup']
 
 self.events = self._dic.get('events',())
 self.categories = self._dic.get('offerCategories',())
class Gamelist:
 DISPLAY_NAME = 'Gamelist'
 
 def __init__(self, events):
 self.events = events
 def to_dict(self):
 return [record for record in self.get_records()]
 def get_records(self):
 for event in self.events:
 if "@" in event['name']:
 yield from self.get_data(event)
 def get_data(self, event):
 book_id = endpoints.gamelist_book_id(event)
 away_abbr = endpoints.gamelist_away_abbr(event)
 home_abbr = endpoints.gamelist_home_abbr(event)
 game_ref = f'{away_abbr}@{home_abbr}'
 game_datetime = endpoints.gamelist_datetime(event)
 data = [book_id, game_ref, game_datetime]
 columns = ['book_id', 'game_ref', 'game_datetime'] 
 return [dict(zip(columns,data))]
class Category:
 def __init__(self, dic, category, subcategory, display_name):
 self.dic = dic
 self.category = category
 self.subcategory = subcategory
 self.display_name = display_name
 self.data = tuple(self.get_category())
 def get_category(self):
 for item in self.dic:
 if item['offerCategoryId'] == self.category:
 yield from self.get_subcategory_description(item['offerSubcategoryDescriptors'])
 def get_subcategory_description(self, data):
 for item in data:
 if item['subcategoryId'] == self.subcategory:
 yield from self.get_subcategory(item['offerSubcategory'].get('offers'))
 def get_subcategory(self, data):
 for item in data:
 for x in item:
 if self.display_name in x.get('label',()) and x.get('isOpen',()):
 yield x
 
class Runline:
 CATEGORY = 493
 SUBCATEGORY = 4519
 DISPLAY_NAME = 'Run Line'
 def __init__(self,data):
 self.data = Category(data, self.CATEGORY, self.SUBCATEGORY, self.DISPLAY_NAME).data
 def to_dict(self):
 return [record for record in self.get_records()]
 
 def get_records(self):
 for market in self.data:
 yield from self.get_data(market)
 def get_data(self, market):
 book_id = endpoints.runline_book_id(market)
 away_abbr = endpoints.runline_away_abbr(market)
 home_abbr = endpoints.runline_home_abbr(market)
 away_odds = endpoints.runline_away_odds(market)
 home_odds = endpoints.runline_home_odds(market)
 home_line = endpoints.runline_home_line(market)
 away = [book_id, away_abbr, away_odds, -home_line, BOOK]
 home = [book_id, home_abbr, home_odds, home_line, BOOK]
 columns = ['book_id', 'team', 'odds', 'line', 'book']
 return [dict(zip(columns,away)), dict(zip(columns,home))]
class Moneyline:
 CATEGORY = 493
 SUBCATEGORY = 4519
 DISPLAY_NAME = 'Moneyline'
 def __init__(self,data):
 self.data = Category(data, self.CATEGORY, self.SUBCATEGORY, self.DISPLAY_NAME).data
 def to_dict(self):
 return [record for record in self.get_records()]
 
 def get_records(self):
 for market in self.data:
 yield from self.get_data(market)
 def get_data(self, market):
 book_id = endpoints.moneyline_book_id(market)
 away_abbr = endpoints.moneyline_away_abbr(market)
 home_abbr = endpoints.moneyline_home_abbr(market)
 away_odds = endpoints.moneyline_away_odds(market)
 home_odds = endpoints.moneyline_home_odds(market)
 away = [book_id, away_abbr, away_odds, BOOK]
 home = [book_id, home_abbr, home_odds, BOOK]
 columns = ['book_id', 'team', 'odds', 'book']
 return [dict(zip(columns,away)), dict(zip(columns,home))]
class Total:
 CATEGORY = 493
 SUBCATEGORY = 4519
 DISPLAY_NAME = 'Total'
 def __init__(self,data):
 self.data = Category(data, self.CATEGORY, self.SUBCATEGORY, self.DISPLAY_NAME).data
 def to_dict(self):
 return [record for record in self.get_records()]
 
 def get_records(self):
 for market in self.data:
 yield from self.get_data(market)
 def get_data(self, market):
 book_id = endpoints.total_book_id(market)
 line = endpoints.total_line(market)
 over = endpoints.total_over(market)
 under = endpoints.total_under(market)
 data = [book_id, line, over, under, BOOK]
 columns = ['book_id', 'line', 'over', 'under', 'book']
 return [dict(zip(columns,data))]
```

Question 2

I believe you've asked this question a couple times before. Thanks for adding more code this time, but it's still unclear what all of this is supposed to do. The title should reflect the purpose of the app, not the programming practice you want reviewed. Reviews are going to be open-ended, so you should be prepared for feedback on all aspects of the application.

Question 3

The purpose is to retrieve data from from multiple json links, parse it and insert to database tables in order to be evaluated. I could really use help on all aspects, so are you saying if I make another post with every script and use a proper title it will be acceptable?

Question 4

No, you can edit this post. A one-sentence description still gives me very little to understand the app. What are "JSON links"? What is the input and result? What's being evaluated? To get good feedback on code, sufficient context is important. Imagine you were given a few hundred lines of code with that sentence. Would you be able to meaningfully review it if you had no prior context? How can I even run this? It's a bunch of classes but no client code that uses the classes, no main.

Question 5

I completely understand. There is a reason why I've been vague but since I really need help I'll provide more context. Thanks for the explanation.

Question 6

@ggorlen I provided further details and added more relevant portions of the project. Please let me know if there is anything else I need to do.

Question 7

Just... yikes, where to begin.

Add PEP484 typehints. If you don't know what they are or why they're beneficial, do some Googling; but in short - a signature like def to_dict(self, markets): tells us nothing about what it actually accepts and only a guess as to what it returns. Type hints don't fix Python's broken type system (they never will), but they are a great deal better than nothing.

Further to the above, a signature like def __init__(self, listofdics, db_control) accepts a listofdics (presumably : list[dict[???, ???]]). That tells us some - not enough - information about the type, and absolutely nothing about the contents. Is this games (a list of game dictionaries)?

GamesDict as a class is not well-named. It isn't a dict. Call it perhaps GamesRepo, if that's what it actually is.

Grammar - it's stat type -> its stat type; it's corresponding table -> its corresponding table.

The OOP strategy followed in GamesDict is inappropriate. Your export_* functions set members such as self.gamelist that should actually only be local variables. Also, the sequence of calls to these export functions should not be done in the constructor; it should be a separate function called from the outside.

Parser and its subclasses are also poor OOP: take _get_records_, _get_subcatdes and _get_subcat. These are all classmethods that accept the same parameter, data. It smells like data should be an instance member, and those should be parameter-less instance methods that refer to data through self.

There is significant code repetition between Runline._get_data and Moneyline._get_data. This suggests that the common code should be pulled out to a common ancestor of these classes.

A class like Total seems to have no actual instance data, and its _get_data returns an anonymous tuple. A more sensible pattern would be for _get_data to instantiate the class via cls(), with the anonymous tuple converted into a class instance with named parameters. By example,

@dataclass
class Total:
 """
 Class Total is a dedicated parser for total statistics. 
 """
 event_id: ???
 line: ???
 over: ???
 under: ???
 book: ???
 
 CATEGORY: ClassVar[int] = 493
 SUBCATEGORY: ClassVar[int] = 4519
 DISPLAY_NAME: ClassVar[str] = 'Total'
 
 @classmethod
 def _get_data(cls, market: dict[str, Any]) -> 'Total':
 event_id = market['eventId']
 
 away_team, home_team = market['outcomes'][:2]
 line = away_team['line']
 over = away_team['oddsDecimal']
 under = home_team['oddsDecimal']
 return cls(
 (event_id, line, over, under, BOOK),
 )

DBControl fails to close its connection and session. Convert it into a context manager whose __exit__ closes all resources that can be closed.

You fail to show where scoped_session is imported from. I assume that this is sqlalchemy.orm.scoping.scoped_session.

Your check_table pattern is fairly broken. SQLAlchemy is a complicated framework with a million and one ways for a beginner to shoot themselves in the foot. Your *Control classes should be converted into ORM classes that inherit from the declarative base. I strongly encourage you to do more reading and follow along with the official tutorial. When the dust settles, a query like this:

 meta = MetaData(self.db_control.engine)
 t1 = Table('mlb_gamelist', meta, autoload=True)
 t2 = Table('caesars_gamelist', meta, autoload=True)
 t3 = Table('fanduel_gamelist', meta, autoload=True)
 t4 = Table('draftkings_gamelist', meta, autoload=True)
 t5 = Table('pointsbet_gamelist', meta, autoload=True)
 t6 = Table('mgm_gamelist', meta, autoload=True)
 sess = self.session()
 data = sess.query(
 t1.c.event_id,
 t2.c.event_id,
 t3.c.event_id,
 t4.c.event_id,
 t5.c.event_id,
 t6.c.event_id
 ).join(
 t2
 ).join(
# ...

must not re-declare the referenced Tables, and instead reference your ORM classes.

Delete sleep(1).

Question 8

Thanks for the response. This is my first attempt at writing something on a larger scale so I knew there would be plenty to critique. I have a few questions but my first would be pertaining to Parser being poor OOP. I understand the approach you suggest but am confused as to how I can pivot to instance methods while still being able to access the CATEGORY values in each subclass. Does that make sense?

Question 9

Statics like CATEGORY are still accessible through self

Question 10

So I ended up going to a method of parse_games without any inheritance and posted the revision. At this stage in the learning process the code makes more sense to me but obviously there is a lot more repetitive code. I had tried implementing your approach as far as _get_data instantiating the class but to be honest that left me even more confused. Could you elaborate on that or maybe provide a brief example?

Question 11

Your example makes sense but I'm confused as to what I should do with the get_records function. Or maybe I better way if explaining is that I'm not sure where to implement a loop that returns the data for each individual game/record.

Reinderien Reinderien 70.9k5 gold badges76 silver badges256 bronze badges · Accepted Answer · 2022-08-28 14:38:37Z