Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Commit f80a821

Browse files
Merge pull request #48 from oracle-samples/carl-downs-batch-1
initial check in of oci-monitoring-metrics-to-datadog-python
2 parents 29c490a + 354e3a3 commit f80a821

File tree

10 files changed

+595
-0
lines changed

10 files changed

+595
-0
lines changed
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
.DS_Store
2+
.project
3+
.classpath
4+
.settings
5+
target
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
Copyright (c) 2022, Oracle and/or its affiliates. All rights reserved.
2+
3+
The Universal Permissive License (UPL), Version 1.0
4+
5+
Subject to the condition set forth below, permission is hereby granted to any person obtaining a copy of this
6+
software, associated documentation and/or data (collectively the "Software"), free of charge and under any
7+
and all copyright rights in the Software, and any and all patent rights owned or freely licensable by each
8+
licensor hereunder covering either (i) the unmodified Software as contributed to or provided by such licensor,
9+
or (ii) the Larger Works (as defined below), to deal in both
10+
11+
(a) the Software, and
12+
13+
(b) any piece of software and/or hardware listed in the lrgrwrks.txt file if one is included with the
14+
Software (each a "Larger Work" to which the Software is contributed by such licensors), without restriction,
15+
including without limitation the rights to copy, create derivative works of, display, perform, and
16+
distribute the Software and make, use, sell, offer for sale, import, export, have made, and have sold
17+
the Software and the Larger Work(s), and to sublicense the foregoing rights on either these or other terms.
18+
19+
This license is subject to the following condition:
20+
21+
The above copyright notice and either this complete permission notice or at a minimum a reference to the
22+
UPL must be included in all copies or substantial portions of the Software.
23+
24+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT
25+
LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
26+
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
27+
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE
28+
OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Lines changed: 232 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,232 @@
1+
# Exporting OCI Monitoring Service Metrics to Datadog
2+
3+
---
4+
5+
## Overview
6+
7+
Let's take a look at bringing Oracle Cloud Infrastructure (OCI)’s rich Metrics resources over to
8+
Datadog to accomplish common goals such DevOps monitoring, application performance monitoring, and so on.
9+
Datadog’s API exposes some unique features. Their API allows you to characterize each metric using tags,
10+
which is essential to aggregating and correlating data as needed for its monitoring, reporting, dashboards, and so on.
11+
12+
Please see the
13+
[companion blog](https://blogs.oracle.com/cloud-infrastructure/post/exporting-oci-monitoring-service-metrics-to-datadog) for more details.
14+
15+
### Prerequisites
16+
17+
If you’re new to Functions, get familiar by running through
18+
the [Quick Start guide on OCI Functions](http://docs.oracle.com/en-us/iaas/Content/Functions/Tasks/functionsquickstartguidestop.htm) before proceeding.
19+
20+
---
21+
## Solution Architecture
22+
23+
![](images/architecture.png)
24+
25+
Here is the basic architecture and flow of data from beginning to end:
26+
27+
* OCI services emit metric data which is captured by the Monitoring service.
28+
* The Monitoring Service feeds metric data events to a Service Connector.
29+
* The Service Connector invokes a Function which transforms the metric data payload to Datadog format and posts the transformed payload to the Datadog REST API.
30+
* Datadog ingests the metrics, building its own aggregations using the provided tagging.
31+
32+
Let's drill down into the OCI Services involved.
33+
34+
---
35+
## Monitoring Service
36+
37+
The [Monitoring Service](https://docs.oracle.com/en-us/iaas/Content/Monitoring/Concepts/monitoringoverview.htm)
38+
receives timestamp-value pairs (aka metric data points) which also carry contextual
39+
dimensions and metadata about the services or applications that emitted them.
40+
41+
---
42+
## Service Connector Hub
43+
44+
The stream of Metric data is event-driven and must be handled on-demand and at scale. The
45+
[Service Connector Hub](https://docs.oracle.com/en-us/iaas/Content/service-connector-hub/overview.htm) does
46+
exactly that. See [Service Connector Hub documentation](https://docs.oracle.com/en-us/iaas/Content/service-connector-hub/overview.htm) for details.
47+
48+
---
49+
## Functions Service
50+
51+
I need to transform between the raw metrics formats and some way to make the Datadog API calls. The
52+
[OCI Functions Service](http://docs.oracle.com/en-us/iaas/Content/Functions/Concepts/functionsoverview.htm) is a
53+
natural fit for the task. Functions integrate nicely with Service Connector Hub as as a target and can scale up
54+
depending on the demand. That lets me focus on writing the logic needed without needing to address how to
55+
deploy and scale it.
56+
57+
---
58+
## Mapping From OCI to DataDog Formats
59+
60+
A key requirement of course is the mapping of OCI to Datadog format. Let's compare the OCI and Datadog
61+
message payload formats, what the mapping needs to accomplish, and see what the resulting transformed message
62+
looks like.
63+
64+
Example OCI Metrics Payload:
65+
66+
{
67+
"namespace": "oci_vcn",
68+
"resourceGroup": null,
69+
"compartmentId": "ocid1.compartment.oc1...",
70+
"name": "VnicFromNetworkBytes",
71+
"dimensions": {
72+
"resourceId": "ocid1.vnic.oc1.phx..."
73+
},
74+
"metadata": {
75+
"displayName": "Bytes from Network",
76+
"unit": "bytes"
77+
},
78+
"datapoints": [
79+
{
80+
"timestamp": 1652196912000,
81+
"value": 5780.0,
82+
"count": 1
83+
}
84+
]
85+
}
86+
87+
Example DataDog Metrics Payload:
88+
89+
{
90+
"series": [
91+
{
92+
"metric": "system.load.1",
93+
"type": 0,
94+
"points": [
95+
{
96+
"timestamp": 1636629071,
97+
'value": 1.1
98+
}
99+
],
100+
"tags": [
101+
"test:ExampleSubmitmetricsreturnsPayloadacceptedresponse"
102+
]
103+
}
104+
]
105+
}
106+
107+
Mapping Behavior:
108+
109+
{
110+
"series": [
111+
{
112+
"metric": "{re-characterized OCI namespace and metric name values}",
113+
"type": {mapped_type_enum},
114+
"points": [
115+
{
116+
"timestamp": {datapoint.timestamp},
117+
"value": {datapoint.value}
118+
}
119+
],
120+
"tags": [
121+
"{metrics tag key1:oci payload value}",
122+
"{metrics tag key2:oci payload_value}"
123+
]
124+
}
125+
]
126+
}
127+
128+
Resulting Output:
129+
130+
{
131+
"series": [
132+
{
133+
"metric": "oci.vcn.vnic.from.network.bytes",
134+
"type": 0,
135+
"points": [
136+
{
137+
"timestamp": 1652196912,
138+
"value": 5780.0
139+
}
140+
],
141+
"tags": [
142+
"name:VnicFromNetworkBytes",
143+
"unit:bytes",
144+
"namespace:oci_vcn",
145+
"displayName:Bytes from Network"
146+
]
147+
}
148+
]
149+
}
150+
151+
---
152+
## Policy Setup
153+
154+
You will need
155+
this [IAM policy](https://docs.oracle.com/en-us/iaas/Content/Functions/Tasks/functionscreatingpolicies.htm#Create_Policies_to_Control_Access_to_Network_and_FunctionRelated_Resources)
156+
to authorize the Service Connector to invoke your Function.
157+
158+
allow any-user to use fn-function in compartment id ocid1.compartment.oc1... where all {request.principal.type=’serviceconnector’, request.principal.compartment.id=’ocid1.compartment.oc1...’}
159+
160+
---
161+
## Service Connector Setup
162+
163+
Now let’s set up a simple service connector instance that takes Monitoring sources and passes them to our Function.
164+
165+
Because your Function requires a VCN, you can use that VCN as the metric source to test against. Let's test
166+
with the `oci_vcn` Monitoring namespace because it will quickly generate a lot of useful events.
167+
168+
Select Monitoring as the source and the Function as the target. Configure your source as the
169+
compartment where the VCN resides and select the Monitoring namespace (`oci_vcn`) that you want to
170+
pick up. Select your Application and the Function within it as the target.
171+
172+
<br />
173+
174+
[<img src="images/sch-setup.png" width="800"/>](image.png)
175+
176+
---
177+
## View Metrics In DataDog
178+
179+
When you have the Service Connector configured, metrics appear in Datadog's Metrics Explorer and notebooks
180+
after a few minutes. The following images show the Metrics Explorer and Notebook user interfaces in
181+
Datadog. Your VCN metrics are displayed.
182+
183+
184+
[<img src="images/datadog1.png" width="800"/>](image.png)
185+
186+
<br />
187+
188+
[<img src="images/datadog2.png" width="800"/>](image.png)
189+
190+
---
191+
## Function Environment
192+
193+
Here are the supported Function parameters:
194+
195+
| Environment Variable | Default | Purpose |
196+
| ------------- |:-------------:| :----- |
197+
| DATADOG_METRICS_API_ENDPOINT | not-configured | REST API endpoint for reaching DataDog ([see docs](https://docs.datadoghq.com/api/latest/metrics/#submit-metrics))|
198+
| DATADOG_API_TOKEN | not-configured | API license token obtained from DataDog |
199+
| METRICS_TAG_KEYS | name, namespace, displayName, resourceDisplayName, unit | OCI Metric Dimensions and metadata to convert to DataDog Metric Tags |
200+
| LOGGING_LEVEL | INFO | Controls function logging outputs. Choices: INFO, WARN, CRITICAL, ERROR, DEBUG |
201+
| ENABLE_TRACING | False | Enables complete exception stack trace logging |
202+
| FORWARD_TO_DATADOG | True | Determines whether messages are forwarded to DataDog |
203+
204+
---
205+
## Conclusion
206+
207+
You now have a low-maintenance, serverless function that can send raw metrics over to DataDog in
208+
near-real time. I encourage you to experiment with the dimensions and metadata tag mappings
209+
to see which combination works best for your use case.
210+
211+
For more information, see the following resources:
212+
213+
- [DataDog Metrics API Reference](https://docs.datadoghq.com/api/latest/metrics/)
214+
- [DataDog Metrics API / Submit Metrics API contract](https://docs.datadoghq.com/api/latest/metrics/#submit-metrics)
215+
216+
---
217+
## **OCI** Related Workshops
218+
219+
LiveLabs is the place to explore Oracle's products and services using workshops designed to
220+
enhance your experience building and deploying applications on the Cloud and On-Premises.
221+
ur library of workshops cover everything from how to provision the world's first autonomous
222+
database to setting up a webserver on our world class OCI Generation 2 infrastructure,
223+
machine learning and much more. Use your existing Oracle Cloud account,
224+
a [Free Tier](https://www.oracle.com/cloud/free/) account or a LiveLabs Cloud Account to build, test,
225+
and deploy applications on Oracle's Cloud.
226+
227+
Visit [LiveLabs](http://bit.ly/golivelabs) now to get started. Workshops are added weekly, please visit frequently for new content.
228+
229+
---
230+
## License
231+
Copyright (c) 2022, Oracle and/or its affiliates. All rights reserved.
232+
Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.

0 commit comments

Comments
(0)

AltStyle によって変換されたページ (->オリジナル) /