Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

AWS X-Ray Instrumentation #11

Unanswered
michaelcohenunsw asked this question in Q&A
Discussion options

Hi,

I am trying to troubleshoot my background jobs which are timing out most of the time.

I am wondering if anyone has implemented AWS X-Ray with the Lambdakiq hander to aid with troubleshooting a Lambdakiq job?

You must be logged in to vote

Replies: 1 comment 6 replies

Comment options

Hey @michaelcohenunsw, sorry for the slow reply. Busy week!

I've always found X-Ray hard to implement. I've used it in one Rails application and the overhead of adding the segments, etc felt like work that was solved in other vendors without me having to think about it. So rather than writing my own segments and annotations, I would rather install New Relic and get their automatic instrumentation.

That said, if you think your timeouts are related to an AWS or some other service call and you wanted to use X-Ray, you could opt for the simple SDK/Net::HTTP patching (https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-ruby-patching.html) to see what information it turns up.

Another different option would be instrumenting your code using AWS Embedded Metrics (https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format.html) which sounds fancy but it is basically creating your own data points (just like AWS does) by formatting some simple JSON to STDOUT/CloudWatch. This is how we get all the good ActiveJob events data (https://github.com/customink/lambdakiq#observability-with-cloudwatch) for you. This gem (https://github.com/customink/aws-embedded-metrics-customink) can help using it. Very useful fo apps that do not use New Relic and need some good data like (p95, p99) percentiles for sections of your code. Hope that helps.

You must be logged in to vote
6 replies
Comment options

Same, let me know how that goes and what you find out... if anything.

Comment options

The app is running on ECS and the background jobs on Lambda. The app.rb:

require 'aws-sdk-ssm'
stage = if ENV['RAILS_ENV'] == "development"
 "dev"
elsif ENV['RAILS_ENV'] == "production"
 "prod"
else
 ENV['RAILS_ENV']
end
# Get secrets from AWS SSM because we can not pass them in to the lambda function securely using CDK
s3_options = {
 region: ENV['AWS_REGION']
}
ssm_client = Aws::SSM::Client.new(**s3_options)
secrets = {
 MYSQL_USER: "/#{stage}/REDACTED",
 MYSQL_PASSWORD: "/#{stage}/REDACTED",
 EMAIL_HOST_USER: "/#{stage}/REDACTED",
 EMAIL_HOST_PASSWORD: "/#{stage}/REDACTED",
 SECRET_KEY_BASE: "/#{stage}/REDACTED"
}
# Set environment variables using SSM parameters
secrets.each do |env_name, parameter_name|
 resp = ssm_client.get_parameter({
 name: parameter_name,
 with_decryption: true,
 })
 ENV[env_name.to_s] = resp.parameter.value
end
if !ENV['MYSQL_PORT'].empty?
 ENV['MYSQL_PORT'] = "3306"
end
ENV['RAILS_LOG_TO_STDOUT'] = "1"
ENV['DATABASE_URL'] = "mysql2://#{ENV['MYSQL_USER']}:#{ENV['MYSQL_PASSWORD']}@#{ENV['MYSQL_HOST']}:#{ENV['MYSQL_PORT']}/#{ENV['MYSQL_DATABASE']}?pool=5&timeout=5000"
require_relative 'config/application'
require_relative 'config/environment'
require 'json'
require 'logger'
def handler(event:, context:)
 logger = Logger.new($stdout)
 logger.info(event.to_json)
 Lambdakiq.handler(event)
end
Comment options

I wonder if things are timing out because of the SSM gets?

Comment options

Thanks for the suggestion. I will add some more logging around the SSM gets and see what happens...

Comment options

Cool, one common technique is to bundle the configs with the deploy package and remove coupling Lambda init to other services all together. See here. Lamby has a rake task that can help. This is why our starter uses Dotenv too.

https://github.com/customink/lamby-cookiecutter-ci/blob/master/%7B%7Bcookiecutter.project_name%7D%7D/bin/_build#L12-L16

I like simple. But another complex option would be to do something like this too. https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/cache-secrets-using-aws-lambda-extensions.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet

AltStyle によって変換されたページ (->オリジナル) /