MySQL Got timeout reading communication packets when reading federated table

Question

I have two servers (solomon and serenity).
Both servers are Windows Server (2012 & 2016), IIS, PHP (5.6.31), MySQL (5.5).

On solomon I have several federated tables set up that point to tables on serenity. And I have a user set up on solomon specifically for connecting to these federated tables.

Accounts are split over the two servers. Half my user accounts are on solomon, half on serenity. However, the login page for all users is on serenity. This is so that the serenity server can check to see which server this user should be directed to, it then redirects to the password page on either solomon or serenity. So in this way, all users go through serenity, but not all users stay there.

However, there are a few other scripts running on solomon that do check these federated tables on serenity, so even though the user has passed through to solomon, the site is still querying these federated tables and thereby hitting serenity.

As far as I know that shouldn't be a problem, but on serenity I'm getting lots of warnings in mysqlerror.log like this: (username, dbname and ip address obfuscated)

180430 16:21:00 [Warning] Aborted connection 24784 to db: 'dbname_blah' user: 'federated_user_blah' host: 'xxx.xxx.xx.xx' (Got timeout reading communication packets)

On solomon I don't get these warnings. Also 95% of these warnings mention the federated user, even though 99% of my queries are done using a different user. So that's why I suspect it's an issue with the federated tables / user.

I wouldn't be too worried about it except that I've been having lots of problems on serenity and I suspect this might be related. Today my database just crashed and I had to reboot the server. Lots of angry customers.

Other Symptoms

For the last few months I've been running into what appears to be concurrent connection limits, as I have several scripts that loop through many rows of data (typically between 50 and 200) and use ajax to call a send email script for each row, or in another case calculate hours. Sometimes these looping scripts don't execute all the calls in the loop but instead die with 500 errors, in particular when there are many rows in the loop (like more than 100 rows). Smaller loops (like 50 records) usually have no problem.

Yet according to Microsoft there are no longer any concurrent connection limits in IIS.

Timeouts on the website Also sometimes not all of my sql statements are able to finish executing on page load. They just time out and return no rows. Then when I refresh the page usually they load, but if it takes more than a few seconds to load I know there will be some statements timing out.

Timeouts in MySQL Workbench And if I try to execute a long running query in MySQL Workbench it will always time out between 10 to 12 seconds. So if I want to select or delete more than 30,000 rows at a time it will typically time out so I always have to limit the number of rows I'm working with.

Backup fails Also, my backups have been failing using MySQLDump. They always fail when trying to backup a rather large table. The smaller tables are no problem but it can't backup this large table which has about 600 MB of data in 7 million rows.

Idle Threads Several weeks ago the server was running really slow and I realized there were hundreds of idle database threads sticking around and not dying. I sort of solved this problem by setting my wait_timeout to 20 seconds, but that seemed like more of a patch than a solution.

Open files limit I ran into another problem when I added open-files-limit=4096 to my.ini as that was recommended based on another setting that I adjusted in an effort to try to solve these issues I've been having. However adding open-files-limit=4096 was a big mistake as that pretty much killed the server. As soon as I commented that out it started working normally again.

I've "fixed" all these problems so far, but I feel like I'm not addressing the root cause of all these issues, just patching them up.

Bottom line, I know there's something wrong with my server and I suspect it may all be related to this warning I keep seeing in mysqlerror.log. Can anybody shed some light on why I might be getting this warning message? Could it be a dns problem? A permissions problem? A bug in MySQL?

I should point out that my servers are both very busy. Each executing several thousand select statements per second.

My entire my.ini file (I've removed comments to keep it short)

[client]
port=3306
[mysql]
default-character-set=UTF8
[mysqld]
federated
ft_min_word_len = 3
log_warnings = 2
log_error=c:\inetpub\wwwroot\mysqlerror.log
connect_timeout = 40
wait_timeout = 40 
max_allowed_packet = 500M
net_read_timeout = 600
net_write_timeout = 600
port=3306
basedir="C:/Program Files/MySQL/MySQL Server 5.5/"
datadir="C:/ProgramData/MySQL/MySQL Server 5.5/Data/"
character-set-server=UTF8
default-storage-engine=INNODB
sql-mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
max_connections=1500
query_cache_size=50M
table_cache=2000
tmp_table_size=256M
thread_cache_size=64
myisam_max_sort_file_size=100G
myisam_sort_buffer_size=69M
key_buffer_size=384M
read_buffer_size=1MB
read_rnd_buffer_size=256K
sort_buffer_size=256K
innodb_additional_mem_pool_size=3M
innodb_flush_log_at_trx_commit=0
innodb_log_buffer_size=32M
innodb_buffer_pool_size=20G
innodb_log_file_size=256M
innodb_thread_concurrency=66
innodb_log_files_in_group = 2
innodb_log_group_home_dir = "C:/Program Files/MySQL/MySQL Server 5.5/data"
query_cache_type = 1
[mysqld_safe]
log_warnings = 2
log_error=c:\inetpub\wwwroot\mysqlerror.log

SHOW GLOBAL STATUS

Aborted_clients 8050
Aborted_connects 5119
Binlog_cache_disk_use 0
Binlog_cache_use 0
Binlog_stmt_cache_disk_use 0
Binlog_stmt_cache_use 0
Bytes_received 99146740665
Bytes_sent 1312788340530
Com_admin_commands 642
Com_assign_to_keycache 0
Com_alter_db 0
Com_alter_db_upgrade 0
Com_alter_event 0
Com_alter_function 0
Com_alter_procedure 0
Com_alter_server 0
Com_alter_table 0
Com_alter_tablespace 0
Com_analyze 0
Com_begin 0
Com_binlog 0
Com_call_procedure 0
Com_change_db 2451188
Com_change_master 0
Com_check 0
Com_checksum 0
Com_commit 0
Com_create_db 8
Com_create_event 0
Com_create_function 0
Com_create_index 0
Com_create_procedure 0
Com_create_server 0
Com_create_table 5
Com_create_trigger 0
Com_create_udf 0
Com_create_user 0
Com_create_view 0
Com_dealloc_sql 0
Com_delete 2298884
Com_delete_multi 0
Com_do 0
Com_drop_db 0
Com_drop_event 0
Com_drop_function 0
Com_drop_index 0
Com_drop_procedure 0
Com_drop_server 0
Com_drop_table 0
Com_drop_trigger 0
Com_drop_user 0
Com_drop_view 0
Com_empty_query 0
Com_execute_sql 0
Com_flush 0
Com_grant 0
Com_ha_close 0
Com_ha_open 0
Com_ha_read 0
Com_help 0
Com_insert 80433
Com_insert_select 3662
Com_install_plugin 0
Com_kill 143
Com_load 0
Com_lock_tables 796
Com_optimize 0
Com_preload_keys 0
Com_prepare_sql 0
Com_purge 0
Com_purge_before_date 0
Com_release_savepoint 0
Com_rename_table 0
Com_rename_user 0
Com_repair 0
Com_replace 0
Com_replace_select 0
Com_reset 0
Com_resignal 0
Com_revoke 0
Com_revoke_all 0
Com_rollback 0
Com_rollback_to_savepoint 0
Com_savepoint 0
Com_select 28145718
Com_set_option 2449199
Com_signal 0
Com_show_authors 0
Com_show_binlog_events 0
Com_show_binlogs 0
Com_show_charsets 1
Com_show_collations 1
Com_show_contributors 0
Com_show_create_db 0
Com_show_create_event 0
Com_show_create_func 1900
Com_show_create_proc 0
Com_show_create_table 428
Com_show_create_trigger 3
Com_show_databases 622
Com_show_engine_logs 0
Com_show_engine_mutex 0
Com_show_engine_status 0
Com_show_events 0
Com_show_errors 0
Com_show_fields 624
Com_show_function_status 381
Com_show_grants 0
Com_show_keys 12
Com_show_master_status 620
Com_show_open_tables 166496
Com_show_plugins 0
Com_show_privileges 0
Com_show_procedure_status 381
Com_show_processlist 352275
Com_show_profile 0
Com_show_profiles 0
Com_show_relaylog_events 0
Com_show_slave_hosts 0
Com_show_slave_status 0
Com_show_status 33924
Com_show_storage_engines 1
Com_show_table_status 9631732
Com_show_tables 427
Com_show_triggers 481
Com_show_variables 779
Com_show_warnings 0
Com_slave_start 0
Com_slave_stop 0
Com_stmt_close 3543
Com_stmt_execute 3543
Com_stmt_fetch 0
Com_stmt_prepare 3543
Com_stmt_reprepare 0
Com_stmt_reset 0
Com_stmt_send_long_data 0
Com_truncate 0
Com_uninstall_plugin 0
Com_unlock_tables 760
Com_update 190582
Com_update_multi 112
Com_xa_commit 0
Com_xa_end 0
Com_xa_prepare 0
Com_xa_recover 0
Com_xa_rollback 0
Com_xa_start 0
Compression OFF
Connections 4902222
Created_tmp_disk_tables 2044232
Created_tmp_files 16985
Created_tmp_tables 19170929
Delayed_errors 0
Delayed_insert_threads 0
Delayed_writes 0
Flush_commands 1
Handler_commit 30519874
Handler_delete 29441
Handler_discover 0
Handler_prepare 0
Handler_read_first 80100
Handler_read_key 3399084113
Handler_read_last 4209
Handler_read_next 402859087
Handler_read_prev 70204318
Handler_read_rnd 447590174
Handler_read_rnd_next 1384408509
Handler_rollback 30345
Handler_savepoint 0
Handler_savepoint_rollback 0
Handler_update 2708725
Handler_write 954512657
Innodb_buffer_pool_pages_data 165761
Innodb_buffer_pool_bytes_data 2715828224
Innodb_buffer_pool_pages_dirty 12
Innodb_buffer_pool_bytes_dirty 196608
Innodb_buffer_pool_pages_flushed 824631
Innodb_buffer_pool_pages_free 1126267
Innodb_buffer_pool_pages_misc 18692
Innodb_buffer_pool_pages_total 1310720
Innodb_buffer_pool_read_ahead_rnd 0
Innodb_buffer_pool_read_ahead 59353
Innodb_buffer_pool_read_ahead_evicted 0
Innodb_buffer_pool_read_requests 1411093916
Innodb_buffer_pool_reads 102643
Innodb_buffer_pool_wait_free 0
Innodb_buffer_pool_write_requests 2315658
Innodb_data_fsyncs 162458
Innodb_data_pending_fsyncs 0
Innodb_data_pending_reads 0
Innodb_data_pending_writes 0
Innodb_data_read 2682212352
Innodb_data_reads 163586
Innodb_data_writes 967117
Innodb_data_written 1404415488
Innodb_dblwr_pages_written 824631
Innodb_dblwr_writes 25129
Innodb_have_atomic_builtins ON
Innodb_log_waits 0
Innodb_log_write_requests 214508
Innodb_log_writes 97683
Innodb_os_log_fsyncs 112195
Innodb_os_log_pending_fsyncs 0
Innodb_os_log_pending_writes 0
Innodb_os_log_written 145280512
Innodb_page_size 16384
Innodb_pages_created 2314
Innodb_pages_read 163447
Innodb_pages_written 824631
Innodb_row_lock_current_waits 0
Innodb_row_lock_time 48
Innodb_row_lock_time_avg 0
Innodb_row_lock_time_max 15
Innodb_row_lock_waits 59
Innodb_rows_deleted 29441
Innodb_rows_inserted 83161
Innodb_rows_read 3180508723
Innodb_rows_updated 138346
Innodb_truncated_status_writes 0
Key_blocks_not_flushed 0
Key_blocks_unused 319666
Key_blocks_used 36
Key_read_requests 2183276
Key_reads 24
Key_write_requests 28603
Key_writes 0
Last_query_cost 0.000000
Max_used_connections 243
Not_flushed_delayed_rows 0
Open_files 2
Open_streams 0
Open_table_definitions 155
Open_tables 116
Opened_files 8196208
Opened_table_definitions 162
Opened_tables 12879
Performance_schema_cond_classes_lost 0
Performance_schema_cond_instances_lost 0
Performance_schema_file_classes_lost 0
Performance_schema_file_handles_lost 0
Performance_schema_file_instances_lost 0
Performance_schema_locker_lost 0
Performance_schema_mutex_classes_lost 0
Performance_schema_mutex_instances_lost 0
Performance_schema_rwlock_classes_lost 0
Performance_schema_rwlock_instances_lost 0
Performance_schema_table_handles_lost 0
Performance_schema_table_instances_lost 0
Performance_schema_thread_classes_lost 0
Performance_schema_thread_instances_lost 0
Prepared_stmt_count 0
Qcache_free_blocks 5902
Qcache_free_memory 17741584
Qcache_hits 631937988
Qcache_inserts 27096342
Qcache_lowmem_prunes 4224427
Qcache_not_cached 1048077
Qcache_queries_in_cache 12188
Qcache_total_blocks 32437
Queries 683199489
Questions 683165600
Rpl_status AUTH_MASTER
Select_full_join 873441
Select_full_range_join 1591
Select_range 18162680
Select_range_check 0
Select_scan 10081022
Slave_heartbeat_period 0.000
Slave_open_temp_tables 0
Slave_received_heartbeats 0
Slave_retried_transactions 0
Slave_running OFF
Slow_launch_threads 0
Slow_queries 174
Sort_merge_passes 15942
Sort_range 1874836
Sort_rows 91682277
Sort_scan 7671476
Ssl_accept_renegotiates 0
Ssl_accepts 0
Ssl_callback_cache_hits 0
Ssl_cipher 
Ssl_cipher_list 
Ssl_client_connects 0
Ssl_connect_renegotiates 0
Ssl_ctx_verify_depth 0
Ssl_ctx_verify_mode 0
Ssl_default_timeout 0
Ssl_finished_accepts 0
Ssl_finished_connects 0
Ssl_session_cache_hits 0
Ssl_session_cache_misses 0
Ssl_session_cache_mode NONE
Ssl_session_cache_overflows 0
Ssl_session_cache_size 0
Ssl_session_cache_timeouts 0
Ssl_sessions_reused 0
Ssl_used_session_cache_entries 0
Ssl_verify_depth 0
Ssl_verify_mode 0
Ssl_version 
Table_locks_immediate 63848411
Table_locks_waited 4
Tc_log_max_pages_used 0
Tc_log_page_size 0
Tc_log_page_waits 0
Threads_cached 63
Threads_connected 14
Threads_created 1124
Threads_running 1
Uptime 186389
Uptime_since_flush_status 186389

SHOW GLOBAL VARIABLES;

auto_increment_increment 1
auto_increment_offset 1
autocommit ON
automatic_sp_privileges ON
back_log 50
basedir C:/Program Files/MySQL/MySQL Server 5.5/
big_tables OFF
binlog_cache_size 32768
binlog_direct_non_transactional_updates OFF
binlog_format STATEMENT
binlog_stmt_cache_size 32768
bulk_insert_buffer_size 8388608
character_set_client utf8
character_set_connection utf8
character_set_database utf8
character_set_filesystem binary
character_set_results utf8
character_set_server utf8
character_set_system utf8
character_sets_dir C:\Program Files\MySQL\MySQL Server 5.5\share\charsets\
collation_connection utf8_general_ci
collation_database utf8_general_ci
collation_server utf8_general_ci
completion_type NO_CHAIN
concurrent_insert AUTO
connect_timeout 40
datadir C:\ProgramData\MySQL\MySQL Server 5.5\Data\
date_format %Y-%m-%d
datetime_format %Y-%m-%d %H:%i:%s
default_storage_engine InnoDB
default_week_format 0
delay_key_write ON
delayed_insert_limit 100
delayed_insert_timeout 300
delayed_queue_size 1000
div_precision_increment 4
engine_condition_pushdown ON
event_scheduler OFF
expire_logs_days 0
flush OFF
flush_time 1800
foreign_key_checks ON
ft_boolean_syntax + -><()~*:""&|
ft_max_word_len 84
ft_min_word_len 3
ft_query_expansion_limit 20
ft_stopword_file (built-in)
general_log OFF
general_log_file C:\ProgramData\MySQL\MySQL Server 5.5\Data\Serenity.log
group_concat_max_len 1024
have_compress YES
have_crypt NO
have_csv YES
have_dynamic_loading YES
have_geometry YES
have_innodb YES
have_ndbcluster NO
have_openssl DISABLED
have_partitioning YES
have_profiling YES
have_query_cache YES
have_rtree_keys YES
have_ssl DISABLED
have_symlink YES
hostname Serenity
ignore_builtin_innodb OFF
init_connect 
init_file 
init_slave 
innodb_adaptive_flushing ON
innodb_adaptive_hash_index ON
innodb_additional_mem_pool_size 3145728
innodb_autoextend_increment 8
innodb_autoinc_lock_mode 1
innodb_buffer_pool_instances 1
innodb_buffer_pool_size 21474836480
innodb_change_buffering all
innodb_checksums ON
innodb_commit_concurrency 0
innodb_concurrency_tickets 500
innodb_data_file_path ibdata1:10M:autoextend
innodb_data_home_dir 
innodb_doublewrite ON
innodb_fast_shutdown 1
innodb_file_format Antelope
innodb_file_format_check ON
innodb_file_format_max Antelope
innodb_file_per_table OFF
innodb_flush_log_at_trx_commit 0
innodb_flush_method 
innodb_force_load_corrupted OFF
innodb_force_recovery 0
innodb_io_capacity 200
innodb_large_prefix OFF
innodb_lock_wait_timeout 50
innodb_locks_unsafe_for_binlog OFF
innodb_log_buffer_size 33554432
innodb_log_file_size 268435456
innodb_log_files_in_group 2
innodb_log_group_home_dir C:\Program Files\MySQL\MySQL Server 5.5\data
innodb_max_dirty_pages_pct 75
innodb_max_purge_lag 0
innodb_mirrored_log_groups 1
innodb_old_blocks_pct 37
innodb_old_blocks_time 0
innodb_open_files 300
innodb_print_all_deadlocks OFF
innodb_purge_batch_size 20
innodb_purge_threads 0
innodb_random_read_ahead OFF
innodb_read_ahead_threshold 56
innodb_read_io_threads 4
innodb_replication_delay 0
innodb_rollback_on_timeout OFF
innodb_rollback_segments 128
innodb_spin_wait_delay 6
innodb_stats_method nulls_equal
innodb_stats_on_metadata ON
innodb_stats_sample_pages 8
innodb_strict_mode OFF
innodb_support_xa ON
innodb_sync_spin_loops 30
innodb_table_locks ON
innodb_thread_concurrency 66
innodb_thread_sleep_delay 10000
innodb_use_native_aio ON
innodb_use_sys_malloc ON
innodb_version 5.5.45
innodb_write_io_threads 4
interactive_timeout 28800
join_buffer_size 131072
keep_files_on_create OFF
key_buffer_size 402653184
key_cache_age_threshold 300
key_cache_block_size 1024
key_cache_division_limit 100
large_files_support ON
large_page_size 0
large_pages OFF
lc_messages en_US
lc_messages_dir C:\Program Files\MySQL\MySQL Server 5.5\share\
lc_time_names en_US
license GPL
local_infile ON
lock_wait_timeout 31536000
log OFF
log_bin OFF
log_bin_trust_function_creators OFF
log_error c:\inetpub\wwwroot\mysqlerror.log
log_output FILE
log_queries_not_using_indexes OFF
log_slave_updates OFF
log_slow_queries OFF
log_warnings 2
long_query_time 10.000000
low_priority_updates OFF
lower_case_file_system ON
lower_case_table_names 1
max_allowed_packet 524288000
max_binlog_cache_size 18446744073709547520
max_binlog_size 1073741824
max_binlog_stmt_cache_size 18446744073709547520
max_connect_errors 10
max_connections 1500
max_delayed_threads 20
max_error_count 64
max_heap_table_size 16777216
max_insert_delayed_threads 20
max_join_size 18446744073709551615
max_length_for_sort_data 1024
max_long_data_size 524288000
max_prepared_stmt_count 16382
max_relay_log_size 0
max_seeks_for_key 4294967295
max_sort_length 1024
max_sp_recursion_depth 0
max_tmp_tables 32
max_user_connections 0
max_write_lock_count 4294967295
metadata_locks_cache_size 1024
min_examined_row_limit 0
multi_range_count 256
myisam_data_pointer_size 6
myisam_max_sort_file_size 107374182400
myisam_mmap_size 18446744073709551615
myisam_recover_options OFF
myisam_repair_threads 1
myisam_sort_buffer_size 72351744
myisam_stats_method nulls_unequal
myisam_use_mmap OFF
named_pipe OFF
net_buffer_length 16384
net_read_timeout 600
net_retry_count 10
net_write_timeout 600
new OFF
old OFF
old_alter_table OFF
old_passwords OFF
open_files_limit 9548
optimizer_prune_level 1
optimizer_search_depth 62
optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on
performance_schema OFF
performance_schema_events_waits_history_long_size 10000
performance_schema_events_waits_history_size 10
performance_schema_max_cond_classes 80
performance_schema_max_cond_instances 1000
performance_schema_max_file_classes 50
performance_schema_max_file_handles 32768
performance_schema_max_file_instances 10000
performance_schema_max_mutex_classes 200
performance_schema_max_mutex_instances 1000000
performance_schema_max_rwlock_classes 30
performance_schema_max_rwlock_instances 1000000
performance_schema_max_table_handles 100000
performance_schema_max_table_instances 50000
performance_schema_max_thread_classes 50
performance_schema_max_thread_instances 1000
pid_file C:\ProgramData\MySQL\MySQL Server 5.5\Data\Serenity.pid
plugin_dir C:\Program Files\MySQL\MySQL Server 5.5\lib\plugin\
port 3306
preload_buffer_size 32768
profiling OFF
profiling_history_size 15
protocol_version 10
query_alloc_block_size 8192
query_cache_limit 1048576
query_cache_min_res_unit 4096
query_cache_size 52428800
query_cache_type ON
query_cache_wlock_invalidate OFF
query_prealloc_size 8192
range_alloc_block_size 4096
read_buffer_size 1048576
read_only OFF
read_rnd_buffer_size 262144
relay_log 
relay_log_index 
relay_log_info_file relay-log.info
relay_log_purge ON
relay_log_recovery OFF
relay_log_space_limit 0
report_host 
report_password 
report_port 3306
report_user 
rpl_recovery_rank 0
secure_auth OFF
secure_file_priv 
server_id 0
shared_memory OFF
shared_memory_base_name MYSQL
skip_external_locking ON
skip_name_resolve OFF
skip_networking OFF
skip_show_database OFF
slave_compressed_protocol OFF
slave_exec_mode STRICT
slave_load_tmpdir C:\Windows\TEMP
slave_max_allowed_packet 1073741824
slave_net_timeout 3600
slave_skip_errors OFF
slave_transaction_retries 10
slave_type_conversions 
slow_launch_time 2
slow_query_log OFF
slow_query_log_file C:\ProgramData\MySQL\MySQL Server 5.5\Data\Serenity-slow.log
socket MySQL
sort_buffer_size 262144
sql_auto_is_null OFF
sql_big_selects ON
sql_big_tables OFF
sql_buffer_result OFF
sql_log_bin ON
sql_log_off OFF
sql_low_priority_updates OFF
sql_max_join_size 18446744073709551615
sql_mode STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
sql_notes ON
sql_quote_show_create ON
sql_safe_updates OFF
sql_select_limit 18446744073709551615
sql_slave_skip_counter 0
sql_warnings OFF
ssl_ca 
ssl_capath 
ssl_cert 
ssl_cipher 
ssl_key 
storage_engine InnoDB
stored_program_cache 256
sync_binlog 0
sync_frm ON
sync_master_info 0
sync_relay_log 0
sync_relay_log_info 0
system_time_zone Eastern Daylight Time
table_definition_cache 400
table_open_cache 2000
thread_cache_size 64
thread_concurrency 10
thread_handling one-thread-per-connection
thread_stack 262144
time_format %H:%i:%s
time_zone SYSTEM
timed_mutexes OFF
tmp_table_size 268435456
tmpdir C:\Windows\TEMP
transaction_alloc_block_size 8192
transaction_prealloc_size 4096
tx_isolation REPEATABLE-READ
unique_checks ON
updatable_views_with_limit YES
version 5.5.45
version_comment MySQL Community Server (GPL)
version_compile_machine x86
version_compile_os Win64
wait_timeout 40

SHOW ENGINE INNODB STATUS;

Didn't fit in this post so I posted it in pastebin: https://pastebin.com/raw/GiJB8rCg

SOLOMON

The following are from Solomon. All these solomon values were taken on Saturday mid day, a time when there is very little traffic on the server.

Solomon my.ini https://pastebin.com/raw/Y9SjtPWv

Solomon Show Global Status https://pastebin.com/raw/WJArViTq

Solomon Show Global Variables https://pastebin.com/raw/ff6t5euN

Solomon Show Engine InnoDB Status https://pastebin.com/raw/J2Yy54uF

• Wilson Hauck: Please post the following (here or on pastebin.com), A) complete my.cnf-ini, complete text results of B) SHOW GLOBAL STATUS; C) SHOW GLOBAL VARIABLES; D) SHOW ENGINE INNODB STATUS; and someone will be able to assist.
• Vincent: Thanks for your reply! I've edited my question to include the information you suggested. Hopefully there's something obvious in there for somebody.
• Wilson Hauck: How much RAM is on Solomon and serenity? Any SSD devices? # CPU's/cores?
• Vincent: Solomon has 64GB Ram with 16 cores. Serenity has 128GB Ram with 20 cores. No SSD drives.
• Vincent: I've added paste bin links to all the Solomon values as requested.

Answer 1

I realize now that this error occurs when idle threads are killed by wait_timeout. In other words, if wait_timeout is set to 40 and idle threads stick around up to 40 seconds, as soon as they hit 40 seconds and get killed off the error is logged: "got timeout reading communication packets".

Normally, the way to avoid threads getting killed by wait_timeout is to call mysql_close() in scripts when the connection is no longer needed.

Unfortunately that doesn't work for queries made through federated tables. So when a query is executed on a federated table, the connection it creates on the other server does not get killed by mysql_close(). (Which is why I have so many idle connections, and why they're only getting killed by wait_timeout)

Answer 2

Observations:

• Version: 5.5.45
• 64 GB of RAM
• Uptime = 2d 03:46:29
• You are running on Windows.
• Running 64-bit version
• You appear to be running entirely (or mostly) InnoDB.

The More Important Issues:

key_buffer_size = 20M
innodb_buffer_pool_instances = 16
long_query_time = 1
max_heap_table_size = 256M
thread_cache_size = 0
join_buffer_size = 1M

It sounds like 64GB is overkill for this dataset. If the dataset grows significantly, set innodb_buffer_pool_size = 40G.

There seem to be some queries that could be improved upon. Turn on the slowlog; wait a day; then see what queries are the "worst". (Several of the Details below point at missing indexes and/or poorly formulated queries.)

There are far more DELETEs than INSERTs; what's up?

CREATE DATABASE about 4 times a day; what's up?

SHOW FUNCTION STATUS several times an hour; what's up?

There are lots of seemingly unnecessary SHOW commands being run; this could be a burden on the system.

Details and other observations:

( Key_blocks_used * 1024 / key_buffer_size ) = 36 * 1024 / 384M = 0.01% -- Percent of key_buffer used. High-water-mark. -- Lower key_buffer_size to avoid unnecessary memory usage.

( innodb_buffer_pool_size / _ram ) = 20480M / 65536M = 31.2% -- % of RAM used for InnoDB buffer_pool

( (key_buffer_size / 0.20 + innodb_buffer_pool_size / 0.70) / _ram ) = (384M / 0.20 + 20480M / 0.70) / 65536M = 47.6% -- Most of available ram should be made available for caching. -- http://mysql.rjweb.org/doc.php/memory

( innodb_buffer_pool_size / innodb_buffer_pool_instances ) = 20480M / 1 = 20480MB -- Size of each buffer_pool instance. -- An instance should be at least 1GB. In very large RAM, have 16 instances.

( Innodb_buffer_pool_pages_free * 16384 / innodb_buffer_pool_size ) = 1,126,267 * 16384 / 20480M = 85.9% -- buffer pool free -- buffer_pool_size is bigger than working set; could decrease it

( Innodb_buffer_pool_pages_free / Innodb_buffer_pool_pages_total ) = 1,126,267 / 1310720 = 85.9% -- Pct of buffer_pool currently not in use -- innodb_buffer_pool_size to bigger than necessary?

( Innodb_pages_written / Innodb_buffer_pool_write_requests ) = 824,631 / 2315658 = 35.6% -- Write requests that had to hit disk -- Check innodb_buffer_pool_size

( Innodb_buffer_pool_bytes_data / innodb_buffer_pool_size ) = 2,715,828,224 / 20480M = 12.6% -- Percent of buffer pool taken up by data -- A small percent may indicate that the buffer_pool is unnecessarily big.

( Innodb_os_log_written / (Uptime / 3600) / innodb_log_files_in_group / innodb_log_file_size ) = 145,280,512 / (186389 / 3600) / 2 / 256M = 0.00523 -- Ratio -- (see minutes)

( Uptime / 60 * innodb_log_file_size / Innodb_os_log_written ) = 186,389 / 60 * 256M / 145280512 = 5,739 -- Minutes between InnoDB log rotations Beginning with 5.6.8, this can be changed dynamically; be sure to also change my.cnf. -- (The recommendation of 60 minutes between rotations is somewhat arbitrary.) Adjust innodb_log_file_size. (Cannot change in AWS.)

( innodb_file_per_table ) = OFF -- Put each file in its own tablespace -- (Mildly recommended, especially for large tables)

( innodb_stats_on_metadata ) = ON -- Re-analyze table when touching stats. -- ON is likely to slow down certain SHOWs and information_schema accesses.

( innodb_thread_concurrency ) = 66 -- 0 = Let InnoDB decide the best for concurrency_tickets. -- Set to 0 or 64. This may cut back on CPU.

( innodb_print_all_deadlocks ) = OFF -- Whether to log all Deadlocks. -- If you are plagued with Deadlocks, turn this on. Caution: If you have lots of deadlocks, this may write a lot to disk.

( net_buffer_length / max_allowed_packet ) = 16,384 / 500M = 0.00%

( local_infile ) = ON -- local_infile = ON is a potential security issue

( bulk_insert_buffer_size / _ram ) = 8M / 65536M = 0.01% -- Buffer for multi-row INSERTs and LOAD DATA -- Too big could threaten RAM size. Too small could hinder such operations.

( Qcache_lowmem_prunes ) = 4,224,427 / 186389 = 23 /sec -- Running out of room in QC -- increase query_cache_size

( Qcache_inserts - Qcache_queries_in_cache ) = (27096342 - 12188) / 186389 = 145 /sec -- Invalidations/sec.

( (query_cache_size - Qcache_free_memory) / Qcache_queries_in_cache / query_alloc_block_size ) = (50M - 17741584) / 12188 / 8192 = 0.347 -- query_alloc_block_size vs formula -- Adjust query_alloc_block_size

( Questions ) = 683,165,600 / 186389 = 3665 /sec -- Queries (outside SP) -- "qps" -->2000 may be stressing server

( Queries ) = 683,199,489 / 186389 = 3665 /sec -- Queries (including inside SP) -->3000 may be stressing server

( Created_tmp_tables ) = 19,170,929 / 186389 = 102 /sec -- Frequency of creating "temp" tables as part of complex SELECTs.

( Created_tmp_disk_tables ) = 2,044,232 / 186389 = 11 /sec -- Frequency of creating disk "temp" tables as part of complex SELECTs -- increase tmp_table_size and max_heap_table_size. Check the rules for temp tables on when MEMORY is used instead of MyISAM. Perhaps minor schema or query changes can avoid MyISAM. Better indexes and reformulation of queries are more likely to help.

( tmp_table_size ) = 256M -- Limit on size of MEMORY temp tables used to support a SELECT -- Decrease tmp_table_size to avoid running out of RAM. Perhaps no more than 64M.

( (Com_insert + Com_update + Com_delete + Com_replace) / Com_commit ) = (80433 + 190582 + 2298884 + 0) / 0 = INF -- Statements per Commit (assuming all InnoDB) -- Low: Might help to group queries together in transactions; High: long transactions strain various things.

( Com_delete / Com_insert ) = 2,298,884 / 80433 = 2858.1% -- Deletes / Inserts (as a pct). (Ignores LOAD, REPLACE, etc.)

( Select_full_join ) = 873,441 / 186389 = 4.7 /sec -- joins without index -- Add suitable index(es) to tables used in JOINs.

( Select_scan ) = 10,081,022 / 186389 = 54 /sec -- full table scans -- Add indexes / optimize queries (unless they are tiny tables)

( Select_scan / Com_select ) = 10,081,022 / 28145718 = 35.8% -- % of selects doing full table scan. (May be fooled by Stored Routines.) -- Add indexes / optimize queries

( log_slow_queries ) = OFF -- Whether to log slow queries. (Before 5.1.29, 5.6.1)

( slow_query_log ) = OFF -- Whether to log slow queries. (5.1.12)

( long_query_time ) = 10.000000 = 10 -- Cutoff (Seconds) for defining a "slow" query. -- Suggest 2

( Com_change_db ) = 2,451,188 / 186389 = 13 /sec -- Probably comes from USE statements. -- Consider connecting with DB, using db.tbl syntax, eliminating spurious USE statements, etc.

( Connections ) = 4,902,222 / 186389 = 26 /sec -- Connections -- Increase wait_timeout; use pooling?

( thread_cache_size ) = 64 -- How many extra processes to keep around (Not relevant when using thread pooling) (Autosized as of 5.6.8; based on max_connections) -- 0 is good for Windows

Abnormally large:

(Com_select + Qcache_hits) / (Com_insert + Com_update + Com_delete + Com_replace) = 256
Bytes_sent = 7043271 /sec
Com_create_db = 0.15 /HR
Com_delete = 12 /sec
Com_kill = 2.8 /HR
Com_show_create_func = 37 /HR
Com_show_databases = 12 /HR
Com_show_function_status = 7.4 /HR
Com_show_master_status = 12 /HR
Com_show_open_tables = 0.89 /sec
Com_show_procedure_status = 7.4 /HR
Com_show_processlist = 1.9 /sec
Com_show_status = 0.18 /sec
Com_show_table_status = 52 /sec
Handler_read_rnd = 2401 /sec
Innodb_buffer_pool_pages_free = 1.13e+6
Qcache_hits = 3390 /sec
Qcache_inserts = 145 /sec
Select_range = 97 /sec
Select_range / Com_select = 64.5%
Sort_scan = 41 /sec
max_long_data_size = 500MB
net_read_timeout = 600
net_write_timeout = 600

Abnormal strings:

ft_min_word_len = 3
have_crypt = NO
innodb_force_load_corrupted = OFF
lower_case_file_system = ON

• Vincent: Rick, thank you for your analysis. You mentioned SHOW commands. There are a lot of show commands happening because I'm using several federated tables and each time a query is run against a federated table mysql executes a "SHOW TABLE STATUS". I'm not sure why it does that, but it does.
• Wilson Hauck: @Vincent I suspect Federated table use require SHOW TABLE STATUS to know how to proceed with tapping Serenity for the current table values on each use. Remember you are still running V 5.5.nn and many processes are better with newer versions.

Answer 3

Add:

wait_timeout=1200
net_read_timeout=1200
net_write_timeout=1200

to your mysql my.cnf and the error should disappear.

• Luís Assunção: This solution solved the problem.

Answer 4

Suggestions to consider for serenity my.cnf-ini in [mysqld] section - no more than one a day and monitor - keep dated copies of each config used for quick replacement, if needed.

wait_timeout=3600 # from 40 seconds may reduce handler_rollback count
innodb_concurrency_tickets=5000 # from 500 reduce getting back in queue
skip_name_resolve=ON # from OFF if not REQUIRED, see refman
read_buffer_size=128K # from 1M to reduce innodb_rows_read
max_connections=300 # from 1500 to reduce RAM footprint max_used_conn was 243
innodb_io_capacity=800 # from default of 200 (unless your media is OLD)
expire_logs_days=5 # from 0 for limited historical logs
key_buffer_size=1M # from 384M only 36 16K blocks used
innodb_buffer_pool_size=10G # from 20G less than 5G used
innodb_read_ahead_threashold=8 # from 56 to RD next extent earlier
max_write_lock_count=16 # from HUGE number to allow RD after nn req
open_files_limit=30000 # from 9548 calc to support 8M opened in 2 days
query_cache_min_res_unit=512 # from 4096 to store more results in QC
slow_query_log=ON # from OFF - minimize ASAP - will take TIME
thread_cache_size=100 # from 64 suggested CAP in v8 refman to avoid OOM
max_heap_table_size=256M # from 16M - should always = tmp_table_size
sort_buffer_size=2M # from 256K - will reduce sort_merge_passes

There will be more analysis and suggestions as progress is observed/measured.

• Vincent: Thanks for this Wilson! I will make these changes one per day and monitor as you suggested.
• Vincent: I set the wait_timeout to 3600 as per your suggestion and I'm no longer getting the "Got timeout reading communication packets" error. Still a few other errors but that one is gone, and my idle threads are not getting out of control.
• Vincent: I'm still getting "Access denied for user ''@'localhost' (using password: NO)" every few minutes. And at 2:00 am when my backup script runs I see a bunch of these errors: "Got an error writing communication packets"
• Vincent: As requested my serenity.ini: pastebin.com/raw/QBsJ0H2T. I didn't bother with the mysqlerror.log because I mentioned in my previous two comments the errors I'm still getting.
• Vincent: Hi Wilson. Didn't get a chance to look at this for a few days, but I did as you suggested and created a new question on that 2:00 am issue. Here is the link: dba.stackexchange.com/questions/206529/…

Answer 5

SET GLOBAL NET_READ_TIMEOUT=3600;

• nbk: please explain always your solution, so that others undeerstand it