-
Notifications
You must be signed in to change notification settings - Fork 8
AWS X-Ray Instrumentation #11
-
Hi,
I am trying to troubleshoot my background jobs which are timing out most of the time.
I am wondering if anyone has implemented AWS X-Ray with the Lambdakiq hander to aid with troubleshooting a Lambdakiq job?
Beta Was this translation helpful? Give feedback.
All reactions
Replies: 1 comment 6 replies
-
Hey @michaelcohenunsw, sorry for the slow reply. Busy week!
I've always found X-Ray hard to implement. I've used it in one Rails application and the overhead of adding the segments, etc felt like work that was solved in other vendors without me having to think about it. So rather than writing my own segments and annotations, I would rather install New Relic and get their automatic instrumentation.
That said, if you think your timeouts are related to an AWS or some other service call and you wanted to use X-Ray, you could opt for the simple SDK/Net::HTTP patching (https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-ruby-patching.html) to see what information it turns up.
Another different option would be instrumenting your code using AWS Embedded Metrics (https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format.html) which sounds fancy but it is basically creating your own data points (just like AWS does) by formatting some simple JSON to STDOUT/CloudWatch. This is how we get all the good ActiveJob events data (https://github.com/customink/lambdakiq#observability-with-cloudwatch) for you. This gem (https://github.com/customink/aws-embedded-metrics-customink) can help using it. Very useful fo apps that do not use New Relic and need some good data like (p95, p99) percentiles for sections of your code. Hope that helps.
Beta Was this translation helpful? Give feedback.
All reactions
-
Same, let me know how that goes and what you find out... if anything.
Beta Was this translation helpful? Give feedback.
All reactions
-
The app is running on ECS and the background jobs on Lambda. The app.rb:
require 'aws-sdk-ssm'
stage = if ENV['RAILS_ENV'] == "development"
"dev"
elsif ENV['RAILS_ENV'] == "production"
"prod"
else
ENV['RAILS_ENV']
end
# Get secrets from AWS SSM because we can not pass them in to the lambda function securely using CDK
s3_options = {
region: ENV['AWS_REGION']
}
ssm_client = Aws::SSM::Client.new(**s3_options)
secrets = {
MYSQL_USER: "/#{stage}/REDACTED",
MYSQL_PASSWORD: "/#{stage}/REDACTED",
EMAIL_HOST_USER: "/#{stage}/REDACTED",
EMAIL_HOST_PASSWORD: "/#{stage}/REDACTED",
SECRET_KEY_BASE: "/#{stage}/REDACTED"
}
# Set environment variables using SSM parameters
secrets.each do |env_name, parameter_name|
resp = ssm_client.get_parameter({
name: parameter_name,
with_decryption: true,
})
ENV[env_name.to_s] = resp.parameter.value
end
if !ENV['MYSQL_PORT'].empty?
ENV['MYSQL_PORT'] = "3306"
end
ENV['RAILS_LOG_TO_STDOUT'] = "1"
ENV['DATABASE_URL'] = "mysql2://#{ENV['MYSQL_USER']}:#{ENV['MYSQL_PASSWORD']}@#{ENV['MYSQL_HOST']}:#{ENV['MYSQL_PORT']}/#{ENV['MYSQL_DATABASE']}?pool=5&timeout=5000"
require_relative 'config/application'
require_relative 'config/environment'
require 'json'
require 'logger'
def handler(event:, context:)
logger = Logger.new($stdout)
logger.info(event.to_json)
Lambdakiq.handler(event)
end
Beta Was this translation helpful? Give feedback.
All reactions
-
I wonder if things are timing out because of the SSM gets?
Beta Was this translation helpful? Give feedback.
All reactions
-
Thanks for the suggestion. I will add some more logging around the SSM gets and see what happens...
Beta Was this translation helpful? Give feedback.
All reactions
-
Cool, one common technique is to bundle the configs with the deploy package and remove coupling Lambda init to other services all together. See here. Lamby has a rake task that can help. This is why our starter uses Dotenv too.
I like simple. But another complex option would be to do something like this too. https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/cache-secrets-using-aws-lambda-extensions.html
Beta Was this translation helpful? Give feedback.