Receive notifications with metadata change feeds
Stay organized with collections
Save and categorize content based on your preferences.
This document describes how to configure Knowledge Catalog (formerly Dataplex Universal Catalog) metadata change feeds to receive notifications through Pub/Sub when metadata within Knowledge Catalog is created, updated, or deleted.
For more information about metadata change feeds, see Metadata change feeds overview.
Before you begin
Familiarize yourself with Pub/Sub and the Dataplex API.
-
Enable the Dataplex API and Pub/Sub APIs.
Roles required to enable APIs
To enable APIs, you need the
serviceusage.services.enablepermission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles. Create a Pub/Sub topic to receive notifications. For more information, see Create a topic.
Install
gcloud. Make sure the short alias ofgcloudis set.Set an alias of
gcurl. This creates a shortcut that includes your authentication token and sets the JSON content-type for API requests:aliasgcurl='curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json"'Set the
DATAPLEX_APIvariable:DATAPLEX_API="dataplex.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION"Replace the following:
PROJECT_ID: your project ID where the Dataplex API is enabledLOCATION: the location where the job runs—for example,us-central1,europe-west3, orasia-south1
Required roles and permissions
To configure metadata change feeds, ensure that you and the Knowledge Catalog service account have the required IAM roles and permissions.
User roles and permissions
To get the permissions that you need to export metadata and access metadata change feeds messages, ask your administrator to grant you the following IAM roles on the project or organization:
-
Export metadata:
Dataplex Entry Group Exporter (
roles/dataplex.entryGroupExporter) -
Access metadata change feeds messages:
Pub/Sub Subscriber (
roles/pubsub.subscriber)
For more information about granting roles, see Manage access to projects, folders, and organizations.
These predefined roles contain the permissions required to export metadata and access metadata change feeds messages. To see the exact permissions that are required, expand the Required permissions section:
Required permissions
The following permissions are required to export metadata and access metadata change feeds messages:
-
Export metadata:
-
dataplex.metadataFeeds.createon the project -
dataplex.entryGroups.exporton the resource defining the scope of the feed (Organization, Project, or Entry Group) -
resourcemanager.projects.geton the project -
resourcemanager.projects.liston the project
-
-
Access metadata change feeds messages:
-
pubsub.subscriptions.consumeon the subscription -
pubsub.topics.attachSubscriptionon the topic
-
You might also be able to get these permissions with custom roles or other predefined roles.
Knowledge Catalog service account roles and permissions
To ensure that the Knowledge Catalog service account has the necessary
permission to publish metadata change feeds messages,
ask your administrator to grant the
Pub/Sub Publisher (roles/pubsub.publisher) and Pub/Sub Viewer (roles/pubsub.viewer)
IAM role to the Knowledge Catalog service account on the Pub/Sub topic.
For more information about granting roles, see Manage access to projects, folders, and organizations.
This predefined role contains the
pubsub.topics.publish and pubsub.topics.get
permission,
which is required to
publish metadata change feeds messages.
Your administrator might also be able to give the Knowledge Catalog service account this permission with custom roles or other predefined roles.
Grant permissions to the Knowledge Catalog service account
The service agent in Knowledge Catalog is created when you enable the Dataplex API. You can identify the service agent by its email ID:
service-PROJECT_NUMBER@gcp-sa-dataplex.iam.gserviceaccount.com
Here, PROJECT_NUMBER is the project
number of the project where you enabled the Dataplex API.
The Knowledge Catalog service account
requires the ability to publish messages to your Pub/Sub topic
and retrieve topic metadata.
To grant these permissions, give the service account the Pub/Sub Publisher role (roles/pubsub.publisher) and the Pub/Sub Viewer role (roles/pubsub.viewer) on the Pub/Sub topic:
gcloud
Run the gcloud pubsub topics add-iam-policy-binding command:
gcloudpubsubtopicsadd-iam-policy-bindingTOPIC_ID\
--member="serviceAccount:service-PROJECT_NUMBER@gcp-sa-dataplex.iam.gserviceaccount.com"\
--role="roles/pubsub.publisher"
gcloudpubsubtopicsadd-iam-policy-bindingTOPIC_ID\
--member="serviceAccount:service-PROJECT_NUMBER@gcp-sa-dataplex.iam.gserviceaccount.com"\
--role="roles/pubsub.viewer"
Replace the following:
TOPIC_ID: the ID of your Pub/Sub topic to which the metadata change feed messages are published.PROJECT_NUMBER: the project number of your Knowledge Catalog project where the service account is located.
Console
In the Google Cloud console, go to the Pub/Sub Topics page.
Select the topic you're using for metadata feed notifications and if required click Show info panel.
In the Permissions tab, click Add principal.
In the New principals field, enter the Knowledge Catalog service account:
service-PROJECT_NUMBER@gcp-sa-dataplex.iam.gserviceaccount.com.In the Assign roles field, select Pub/Sub Publisher.
Click Add another role and select Pub/Sub Viewer.
Click Save.
Create a metadata change feed
To control which changes generate notifications, you can configure a metadata change feed to monitor specific resources. You do this by specifying a scope, such as your entire organization, specific projects, or specific entry groups. While scope lets you define which resources to monitor, you can use filters to further refine when Knowledge Catalog sends notifications.
For more information, see Metadata change feed.
C#
Before trying this sample, follow the C# setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog C# API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
C#
usingGoogle.Api.Gax.ResourceNames ;
usingGoogle.Cloud.Dataplex.V1 ;
usingGoogle.LongRunning ;
publicsealedpartialclassGeneratedCatalogServiceClientSnippets
{
/// <summary>Snippet for CreateMetadataFeed</summary>
/// <remarks>
/// This snippet has been automatically generated and should be regarded as a code template only.
/// It will require modifications to work:
/// - It may require correct/in-range values for request initialization.
/// - It may require specifying regional endpoints when creating the service client as shown in
/// https://cloud.google.com/dotnet/docs/reference/help/client-configuration#endpoint.
/// </remarks>
publicvoidCreateMetadataFeedRequestObject()
{
// Create client
CatalogServiceClient catalogServiceClient=CatalogServiceClient .Create ();
// Initialize request argument(s)
CreateMetadataFeedRequest request=newCreateMetadataFeedRequest
{
ParentAsLocationName=LocationName .FromProjectLocation ("[PROJECT]","[LOCATION]"),
MetadataFeed=newMetadataFeed (),
MetadataFeedId="",
ValidateOnly=false,
};
// Make the request
Operation<MetadataFeed,OperationMetadata>response=catalogServiceClient.CreateMetadataFeed (request);
// Poll until the returned long-running operation is complete
Operation<MetadataFeed,OperationMetadata>completedResponse=response.PollUntilCompleted();
// Retrieve the operation result
MetadataFeed result=completedResponse.Result;
// Or get the name of the operation
stringoperationName=response.Name;
// This name can be stored, then the long-running operation retrieved later by name
Operation<MetadataFeed,OperationMetadata>retrievedResponse=catalogServiceClient.PollOnceCreateMetadataFeed (operationName);
// Check if the retrieved long-running operation has completed
if(retrievedResponse.IsCompleted)
{
// If it has completed, then access the result
MetadataFeed retrievedResult=retrievedResponse.Result;
}
}
}
Go
Before trying this sample, follow the Go setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Go API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Go
//go:build examples
packagemain
import(
"context"
dataplex"cloud.google.com/go/dataplex/apiv1"
dataplexpb"cloud.google.com/go/dataplex/apiv1/dataplexpb"
)
funcmain(){
ctx:=context.Background()
// This snippet has been automatically generated and should be regarded as a code template only.
// It will require modifications to work:
// - It may require correct/in-range values for request initialization.
// - It may require specifying regional endpoints when creating the service client as shown in:
// https://pkg.go.dev/cloud.google.com/go#hdr-Client_Options
c,err:=dataplex.NewCatalogClient (ctx)
iferr!=nil{
// TODO: Handle error.
}
deferc.Close()
req:=&dataplexpb.CreateMetadataFeedRequest{
// TODO: Fill request struct fields.
// See https://pkg.go.dev/cloud.google.com/go/dataplex/apiv1/dataplexpb#CreateMetadataFeedRequest.
}
op,err:=c.CreateMetadataFeed(ctx,req)
iferr!=nil{
// TODO: Handle error.
}
resp,err:=op.Wait(ctx)
iferr!=nil{
// TODO: Handle error.
}
// TODO: Use resp.
_=resp
}
Java
Before trying this sample, follow the Java setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Java API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Java
importcom.google.cloud.dataplex.v1.CatalogServiceClient ;
importcom.google.cloud.dataplex.v1.CreateMetadataFeedRequest ;
importcom.google.cloud.dataplex.v1.LocationName ;
importcom.google.cloud.dataplex.v1.MetadataFeed ;
publicclass SyncCreateMetadataFeed{
publicstaticvoidmain(String[]args)throwsException{
syncCreateMetadataFeed();
}
publicstaticvoidsyncCreateMetadataFeed()throwsException{
// This snippet has been automatically generated and should be regarded as a code template only.
// It will require modifications to work:
// - It may require correct/in-range values for request initialization.
// - It may require specifying regional endpoints when creating the service client as shown in
// https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
try(CatalogServiceClient catalogServiceClient=CatalogServiceClient .create()){
CreateMetadataFeedRequest request=
CreateMetadataFeedRequest .newBuilder()
.setParent(LocationName .of("[PROJECT]","[LOCATION]").toString())
.setMetadataFeed(MetadataFeed .newBuilder().build())
.setMetadataFeedId ("metadataFeedId1633421160")
.setValidateOnly(true)
.build();
MetadataFeed response=catalogServiceClient.createMetadataFeedAsync (request).get();
}
}
}
Node.js
Before trying this sample, follow the Node.js setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Node.js API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Node.js
/**
* This snippet has been automatically generated and should be regarded as a code template only.
* It will require modifications to work.
* It may require correct/in-range values for request initialization.
* TODO(developer): Uncomment these variables before running the sample.
*/
/**
* Required. The resource name of the parent location, in the format
* `projects/{project_id_or_number}/locations/{location_id}`
*/
// const parent = 'abc123'
/**
* Required. The metadata job resource.
*/
// const metadataFeed = {}
/**
* Optional. The metadata job ID. If not provided, a unique ID is generated
* with the prefix `metadata-job-`.
*/
// const metadataFeedId = 'abc123'
/**
* Optional. The service validates the request without performing any
* mutations. The default is false.
*/
// const validateOnly = true
// Imports the Dataplex library
const{CatalogServiceClient}=require('@google-cloud/dataplex').v1;
// Instantiates a client
constdataplexClient=newCatalogServiceClient ();
asyncfunctioncallCreateMetadataFeed(){
// Construct request
constrequest={
parent,
metadataFeed,
};
// Run request
const[operation]=awaitdataplexClient.createMetadataFeed(request);
const[response]=awaitoperation.promise();
console.log(response);
}
callCreateMetadataFeed();
Python
Before trying this sample, follow the Python setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Python API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Python
# This snippet has been automatically generated and should be regarded as a
# code template only.
# It will require modifications to work:
# - It may require correct/in-range values for request initialization.
# - It may require specifying regional endpoints when creating the service
# client as shown in:
# https://googleapis.dev/python/google-api-core/latest/client_options.html
fromgoogle.cloudimport dataplex_v1
defsample_create_metadata_feed():
# Create a client
client = dataplex_v1 .CatalogServiceClient ()
# Initialize request argument(s)
metadata_feed = dataplex_v1 .MetadataFeed ()
metadata_feed.pubsub_topic = "pubsub_topic_value"
request = dataplex_v1 .CreateMetadataFeedRequest (
parent="parent_value",
metadata_feed=metadata_feed,
)
# Make the request
operation = client.create_metadata_feed (request=request)
print("Waiting for operation to complete...")
response = operation.result()
# Handle the response
print(response)
REST
To create a metadata change feed, use the
projects.locations.metadataFeeds.create
method.
Organization scope
Run the following command to create a metadata change feed that monitors the entire organization:
gcurl-XPOST-d"$(cat<<EOF
{
"scope":{
"organizationLevel":true
},
"pubsubTopic":"projects/PROJECT_ID_PUBSUB/topics/TOPIC_ID"
}
EOF
)""https://${DATAPLEX_API}/metadataFeeds?metadataFeedId=FEED_ID"
Replace the following:
PROJECT_ID_PUBSUB: the project ID where the Pub/Sub topic is located—for example,example-projectTOPIC_ID: the Pub/Sub topic ID to which the metadata change feed messages are published—for example,example-topicFEED_ID: the metadata change feed ID that you want to create—for example,example-feed
Project scope
Run the following command to create a metadata change feed that monitors specific projects:
gcurl-XPOST-d"$(cat<<EOF
{
"scope":{
"projects":[
"projects/PROJECT_ID_1",
"projects/PROJECT_ID_2"
]
},
"pubsubTopic":"projects/PROJECT_ID_PUBSUB/topics/TOPIC_ID"
}
EOF
)""https://${DATAPLEX_API}/metadataFeeds?metadataFeedId=FEED_ID"
Replace the following:
PROJECT_ID_PUBSUB: the project ID where the Pub/Sub topic is located—for example,example-projectTOPIC_ID: the Pub/Sub topic ID to which the metadata change feed messages are published—for example,example-topicFEED_ID: the metadata change feed ID that you want to create—for example,example-feed
Entry group scope
Run the following command to create a metadata change feed that monitors specific entry groups:
gcurl-XPOST-d"$(cat<<EOF
{
"scope":{
"entryGroups":[
"projects/PROJECT_ID/locations/LOCATION/entryGroups/ENTRY_GROUP_ID_1",
"projects/PROJECT_ID/locations/LOCATION/entryGroups/ENTRY_GROUP_ID_2"
]
},
"pubsubTopic":"projects/PROJECT_ID_PUBSUB/topics/TOPIC_ID"
}
EOF
)""https://${DATAPLEX_API}/metadataFeeds?metadataFeedId=FEED_ID"
Replace the following:
PROJECT_ID_PUBSUB: the project ID where the Pub/Sub topic is located—for example,example-projectLOCATION: the location where you want to create the feed—for example,us-central1ENTRY_GROUP_ID: the entry group ID to monitor—for example,example-entry-groupTOPIC_ID: the Pub/Sub topic ID to which the metadata change feed messages are published—for example,example-topicFEED_ID: the metadata change feed ID that you want to create—for example,example-feed
With filters
Run the following command to create a metadata change feed with filters (filtering by entry type):
gcurl-XPOST-d"$(cat<<EOF
{
"scope":{
"projects":[
"projects/PROJECT_ID_1",
"projects/PROJECT_ID_2"
]
},
"filter":{
"entryTypes":[
"projects/PROJECT_ID/locations/global/entryTypes/bigquery-table"
]
},
"pubsubTopic":"projects/PROJECT_ID_PUBSUB/topics/TOPIC_ID"
}
EOF
)""https://${DATAPLEX_API}/metadataFeeds?metadataFeedId=FEED_ID"
Replace the following:
PROJECT_ID_PUBSUB: the project ID where the Pub/Sub topic is located—for example,example-projectTOPIC_ID: the Pub/Sub topic ID to which the metadata change feed messages are published—for example,example-topicFEED_ID: the metadata change feed ID that you want to create—for example,example-feed
Response:
{
"name":"projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID",
"metadata":{
"@type":"type.googleapis.com/google.cloud.dataplex.v1.OperationMetadata",
"createTime":"2023-10-02T15:01:23Z",
"target":"projects/PROJECT_ID/locations/LOCATION/metadataFeeds/FEED_ID",
"verb":"create",
"apiVersion":"v1"
},
"done":false
}
View a metadata change feed
You can view the details of a metadata change feed.
C#
Before trying this sample, follow the C# setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog C# API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
C#
usingGoogle.Cloud.Dataplex.V1 ;
publicsealedpartialclassGeneratedCatalogServiceClientSnippets
{
/// <summary>Snippet for GetMetadataFeed</summary>
/// <remarks>
/// This snippet has been automatically generated and should be regarded as a code template only.
/// It will require modifications to work:
/// - It may require correct/in-range values for request initialization.
/// - It may require specifying regional endpoints when creating the service client as shown in
/// https://cloud.google.com/dotnet/docs/reference/help/client-configuration#endpoint.
/// </remarks>
publicvoidGetMetadataFeedRequestObject()
{
// Create client
CatalogServiceClient catalogServiceClient=CatalogServiceClient .Create ();
// Initialize request argument(s)
GetMetadataFeedRequest request=newGetMetadataFeedRequest
{
MetadataFeedName=MetadataFeedName .FromProjectLocationMetadataFeed ("[PROJECT]","[LOCATION]","[METADATA_FEED]"),
};
// Make the request
MetadataFeed response=catalogServiceClient.GetMetadataFeed (request);
}
}
Go
Before trying this sample, follow the Go setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Go API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Go
//go:build examples
packagemain
import(
"context"
dataplex"cloud.google.com/go/dataplex/apiv1"
dataplexpb"cloud.google.com/go/dataplex/apiv1/dataplexpb"
)
funcmain(){
ctx:=context.Background()
// This snippet has been automatically generated and should be regarded as a code template only.
// It will require modifications to work:
// - It may require correct/in-range values for request initialization.
// - It may require specifying regional endpoints when creating the service client as shown in:
// https://pkg.go.dev/cloud.google.com/go#hdr-Client_Options
c,err:=dataplex.NewCatalogClient (ctx)
iferr!=nil{
// TODO: Handle error.
}
deferc.Close()
req:=&dataplexpb.GetMetadataFeedRequest{
// TODO: Fill request struct fields.
// See https://pkg.go.dev/cloud.google.com/go/dataplex/apiv1/dataplexpb#GetMetadataFeedRequest.
}
resp,err:=c.GetMetadataFeed(ctx,req)
iferr!=nil{
// TODO: Handle error.
}
// TODO: Use resp.
_=resp
}
Java
Before trying this sample, follow the Java setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Java API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Java
importcom.google.cloud.dataplex.v1.CatalogServiceClient ;
importcom.google.cloud.dataplex.v1.GetMetadataFeedRequest ;
importcom.google.cloud.dataplex.v1.MetadataFeed ;
importcom.google.cloud.dataplex.v1.MetadataFeedName ;
publicclass SyncGetMetadataFeed{
publicstaticvoidmain(String[]args)throwsException{
syncGetMetadataFeed();
}
publicstaticvoidsyncGetMetadataFeed()throwsException{
// This snippet has been automatically generated and should be regarded as a code template only.
// It will require modifications to work:
// - It may require correct/in-range values for request initialization.
// - It may require specifying regional endpoints when creating the service client as shown in
// https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
try(CatalogServiceClient catalogServiceClient=CatalogServiceClient .create()){
GetMetadataFeedRequest request=
GetMetadataFeedRequest .newBuilder()
.setName(MetadataFeedName .of("[PROJECT]","[LOCATION]","[METADATA_FEED]").toString())
.build();
MetadataFeed response=catalogServiceClient.getMetadataFeed(request);
}
}
}
Node.js
Before trying this sample, follow the Node.js setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Node.js API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Node.js
/**
* This snippet has been automatically generated and should be regarded as a code template only.
* It will require modifications to work.
* It may require correct/in-range values for request initialization.
* TODO(developer): Uncomment these variables before running the sample.
*/
/**
* Required. The resource name of the metadata feed, in the format
* `projects/{project_id_or_number}/locations/{location_id}/MetadataFeeds/{metadata_feed_id}`.
*/
// const name = 'abc123'
// Imports the Dataplex library
const{CatalogServiceClient}=require('@google-cloud/dataplex').v1;
// Instantiates a client
constdataplexClient=newCatalogServiceClient ();
asyncfunctioncallGetMetadataFeed(){
// Construct request
constrequest={
name,
};
// Run request
constresponse=awaitdataplexClient.getMetadataFeed(request);
console.log(response);
}
callGetMetadataFeed();
Python
Before trying this sample, follow the Python setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Python API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Python
# This snippet has been automatically generated and should be regarded as a
# code template only.
# It will require modifications to work:
# - It may require correct/in-range values for request initialization.
# - It may require specifying regional endpoints when creating the service
# client as shown in:
# https://googleapis.dev/python/google-api-core/latest/client_options.html
fromgoogle.cloudimport dataplex_v1
defsample_get_metadata_feed():
# Create a client
client = dataplex_v1 .CatalogServiceClient ()
# Initialize request argument(s)
request = dataplex_v1 .GetMetadataFeedRequest (
name="name_value",
)
# Make the request
response = client.get_metadata_feed (request=request)
# Handle the response
print(response)
REST
To view a metadata change feed, use the
projects.locations.metadataFeeds.get
method.
Run the following command:
gcurl"https://${DATAPLEX_API}/metadataFeeds/FEED_ID"
Replace the following:
FEED_ID: the metadata change feed ID that you want to view—for example,example-feed
List metadata change feeds
You can list the metadata change feeds in a project and location.
C#
Before trying this sample, follow the C# setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog C# API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
C#
usingGoogle.Api.Gax ;
usingGoogle.Api.Gax.ResourceNames ;
usingGoogle.Cloud.Dataplex.V1 ;
usingSystem;
publicsealedpartialclassGeneratedCatalogServiceClientSnippets
{
/// <summary>Snippet for ListMetadataFeeds</summary>
/// <remarks>
/// This snippet has been automatically generated and should be regarded as a code template only.
/// It will require modifications to work:
/// - It may require correct/in-range values for request initialization.
/// - It may require specifying regional endpoints when creating the service client as shown in
/// https://cloud.google.com/dotnet/docs/reference/help/client-configuration#endpoint.
/// </remarks>
publicvoidListMetadataFeedsRequestObject()
{
// Create client
CatalogServiceClient catalogServiceClient=CatalogServiceClient .Create ();
// Initialize request argument(s)
ListMetadataFeedsRequest request=newListMetadataFeedsRequest
{
ParentAsLocationName=LocationName .FromProjectLocation ("[PROJECT]","[LOCATION]"),
Filter="",
OrderBy="",
};
// Make the request
PagedEnumerable<ListMetadataFeedsResponse,MetadataFeed>response=catalogServiceClient.ListMetadataFeeds (request);
// Iterate over all response items, lazily performing RPCs as required
foreach(MetadataFeed iteminresponse)
{
// Do something with each item
Console.WriteLine(item);
}
// Or iterate over pages (of server-defined size), performing one RPC per page
foreach(ListMetadataFeedsResponse pageinresponse.AsRawResponses())
{
// Do something with each page of items
Console.WriteLine("A page of results:");
foreach(MetadataFeed iteminpage)
{
// Do something with each item
Console.WriteLine(item);
}
}
// Or retrieve a single page of known size (unless it's the final page), performing as many RPCs as required
intpageSize=10;
Page<MetadataFeed>singlePage=response.ReadPage(pageSize);
// Do something with the page of items
Console.WriteLine($"A page of {pageSize} results (unless it's the final page):");
foreach(MetadataFeed iteminsinglePage)
{
// Do something with each item
Console.WriteLine(item);
}
// Store the pageToken, for when the next page is required.
stringnextPageToken=singlePage.NextPageToken;
}
}
Go
Before trying this sample, follow the Go setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Go API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Go
//go:build examples
packagemain
import(
"context"
dataplex"cloud.google.com/go/dataplex/apiv1"
dataplexpb"cloud.google.com/go/dataplex/apiv1/dataplexpb"
"google.golang.org/api/iterator"
)
funcmain(){
ctx:=context.Background()
// This snippet has been automatically generated and should be regarded as a code template only.
// It will require modifications to work:
// - It may require correct/in-range values for request initialization.
// - It may require specifying regional endpoints when creating the service client as shown in:
// https://pkg.go.dev/cloud.google.com/go#hdr-Client_Options
c,err:=dataplex.NewCatalogClient (ctx)
iferr!=nil{
// TODO: Handle error.
}
deferc.Close()
req:=&dataplexpb.ListMetadataFeedsRequest{
// TODO: Fill request struct fields.
// See https://pkg.go.dev/cloud.google.com/go/dataplex/apiv1/dataplexpb#ListMetadataFeedsRequest.
}
it:=c.ListMetadataFeeds(ctx,req)
for{
resp,err:=it.Next()
iferr==iterator.Done{
break
}
iferr!=nil{
// TODO: Handle error.
}
// TODO: Use resp.
_=resp
// If you need to access the underlying RPC response,
// you can do so by casting the `Response` as below.
// Otherwise, remove this line. Only populated after
// first call to Next(). Not safe for concurrent access.
_=it.Response.(*dataplexpb.ListMetadataFeedsResponse)
}
}
Java
Before trying this sample, follow the Java setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Java API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Java
importcom.google.cloud.dataplex.v1.CatalogServiceClient ;
importcom.google.cloud.dataplex.v1.ListMetadataFeedsRequest ;
importcom.google.cloud.dataplex.v1.LocationName ;
importcom.google.cloud.dataplex.v1.MetadataFeed ;
publicclass SyncListMetadataFeeds{
publicstaticvoidmain(String[]args)throwsException{
syncListMetadataFeeds();
}
publicstaticvoidsyncListMetadataFeeds()throwsException{
// This snippet has been automatically generated and should be regarded as a code template only.
// It will require modifications to work:
// - It may require correct/in-range values for request initialization.
// - It may require specifying regional endpoints when creating the service client as shown in
// https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
try(CatalogServiceClient catalogServiceClient=CatalogServiceClient .create()){
ListMetadataFeedsRequest request=
ListMetadataFeedsRequest .newBuilder()
.setParent(LocationName .of("[PROJECT]","[LOCATION]").toString())
.setPageSize(883849137)
.setPageToken("pageToken873572522")
.setFilter("filter-1274492040")
.setOrderBy("orderBy-1207110587")
.build();
for(MetadataFeed element:catalogServiceClient.listMetadataFeeds(request).iterateAll()){
// doThingsWith(element);
}
}
}
}
Node.js
Before trying this sample, follow the Node.js setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Node.js API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Node.js
/**
* This snippet has been automatically generated and should be regarded as a code template only.
* It will require modifications to work.
* It may require correct/in-range values for request initialization.
* TODO(developer): Uncomment these variables before running the sample.
*/
/**
* Required. The resource name of the parent location, in the format
* `projects/{project_id_or_number}/locations/{location_id}`
*/
// const parent = 'abc123'
/**
* Optional. The maximum number of metadata feeds to return. The service
* might return fewer feeds than this value. If unspecified, at most 10 feeds
* are returned. The maximum value is 1,000.
*/
// const pageSize = 1234
/**
* Optional. The page token received from a previous `ListMetadataFeeds` call.
* Provide this token to retrieve the subsequent page of results. When
* paginating, all other parameters that are provided to the
* `ListMetadataFeeds` request must match the call that provided the
* page token.
*/
// const pageToken = 'abc123'
/**
* Optional. Filter request. Filters are case-sensitive.
* The service supports the following formats:
* * `labels.key1 = "value1"`
* * `labels:key1`
* * `name = "value"`
* You can combine filters with `AND`, `OR`, and `NOT` operators.
*/
// const filter = 'abc123'
/**
* Optional. The field to sort the results by, either `name` or `create_time`.
* If not specified, the ordering is undefined.
*/
// const orderBy = 'abc123'
// Imports the Dataplex library
const{CatalogServiceClient}=require('@google-cloud/dataplex').v1;
// Instantiates a client
constdataplexClient=newCatalogServiceClient ();
asyncfunctioncallListMetadataFeeds(){
// Construct request
constrequest={
parent,
};
// Run request
constiterable=dataplexClient.listMetadataFeedsAsync (request);
forawait(constresponseofiterable){
console.log(response);
}
}
callListMetadataFeeds();
Python
Before trying this sample, follow the Python setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Python API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Python
# This snippet has been automatically generated and should be regarded as a
# code template only.
# It will require modifications to work:
# - It may require correct/in-range values for request initialization.
# - It may require specifying regional endpoints when creating the service
# client as shown in:
# https://googleapis.dev/python/google-api-core/latest/client_options.html
fromgoogle.cloudimport dataplex_v1
defsample_list_metadata_feeds():
# Create a client
client = dataplex_v1 .CatalogServiceClient ()
# Initialize request argument(s)
request = dataplex_v1 .ListMetadataFeedsRequest (
parent="parent_value",
)
# Make the request
page_result = client.list_metadata_feeds (request=request)
# Handle the response
for response in page_result:
print(response)
REST
To list metadata change feeds, use the
projects.locations.metadataFeeds.list
method.
Run the following command:
gcurl"https://${DATAPLEX_API}/metadataFeeds"
Update a metadata change feed
You can update the scope or filters of an existing metadata change feed.
C#
Before trying this sample, follow the C# setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog C# API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
C#
usingGoogle.Cloud.Dataplex.V1 ;
usingGoogle.LongRunning ;
usingGoogle.Protobuf.WellKnownTypes ;
publicsealedpartialclassGeneratedCatalogServiceClientSnippets
{
/// <summary>Snippet for UpdateMetadataFeed</summary>
/// <remarks>
/// This snippet has been automatically generated and should be regarded as a code template only.
/// It will require modifications to work:
/// - It may require correct/in-range values for request initialization.
/// - It may require specifying regional endpoints when creating the service client as shown in
/// https://cloud.google.com/dotnet/docs/reference/help/client-configuration#endpoint.
/// </remarks>
publicvoidUpdateMetadataFeedRequestObject()
{
// Create client
CatalogServiceClient catalogServiceClient=CatalogServiceClient .Create ();
// Initialize request argument(s)
UpdateMetadataFeedRequest request=newUpdateMetadataFeedRequest
{
MetadataFeed=newMetadataFeed (),
UpdateMask=newFieldMask (),
ValidateOnly=false,
};
// Make the request
Operation<MetadataFeed,OperationMetadata>response=catalogServiceClient.UpdateMetadataFeed (request);
// Poll until the returned long-running operation is complete
Operation<MetadataFeed,OperationMetadata>completedResponse=response.PollUntilCompleted();
// Retrieve the operation result
MetadataFeed result=completedResponse.Result;
// Or get the name of the operation
stringoperationName=response.Name;
// This name can be stored, then the long-running operation retrieved later by name
Operation<MetadataFeed,OperationMetadata>retrievedResponse=catalogServiceClient.PollOnceUpdateMetadataFeed (operationName);
// Check if the retrieved long-running operation has completed
if(retrievedResponse.IsCompleted)
{
// If it has completed, then access the result
MetadataFeed retrievedResult=retrievedResponse.Result;
}
}
}
Go
Before trying this sample, follow the Go setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Go API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Go
//go:build examples
packagemain
import(
"context"
dataplex"cloud.google.com/go/dataplex/apiv1"
dataplexpb"cloud.google.com/go/dataplex/apiv1/dataplexpb"
)
funcmain(){
ctx:=context.Background()
// This snippet has been automatically generated and should be regarded as a code template only.
// It will require modifications to work:
// - It may require correct/in-range values for request initialization.
// - It may require specifying regional endpoints when creating the service client as shown in:
// https://pkg.go.dev/cloud.google.com/go#hdr-Client_Options
c,err:=dataplex.NewCatalogClient (ctx)
iferr!=nil{
// TODO: Handle error.
}
deferc.Close()
req:=&dataplexpb.UpdateMetadataFeedRequest{
// TODO: Fill request struct fields.
// See https://pkg.go.dev/cloud.google.com/go/dataplex/apiv1/dataplexpb#UpdateMetadataFeedRequest.
}
op,err:=c.UpdateMetadataFeed(ctx,req)
iferr!=nil{
// TODO: Handle error.
}
resp,err:=op.Wait(ctx)
iferr!=nil{
// TODO: Handle error.
}
// TODO: Use resp.
_=resp
}
Java
Before trying this sample, follow the Java setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Java API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Java
importcom.google.cloud.dataplex.v1.CatalogServiceClient ;
importcom.google.cloud.dataplex.v1.MetadataFeed ;
importcom.google.cloud.dataplex.v1.UpdateMetadataFeedRequest ;
importcom.google.protobuf.FieldMask ;
publicclass SyncUpdateMetadataFeed{
publicstaticvoidmain(String[]args)throwsException{
syncUpdateMetadataFeed();
}
publicstaticvoidsyncUpdateMetadataFeed()throwsException{
// This snippet has been automatically generated and should be regarded as a code template only.
// It will require modifications to work:
// - It may require correct/in-range values for request initialization.
// - It may require specifying regional endpoints when creating the service client as shown in
// https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
try(CatalogServiceClient catalogServiceClient=CatalogServiceClient .create()){
UpdateMetadataFeedRequest request=
UpdateMetadataFeedRequest .newBuilder()
.setMetadataFeed(MetadataFeed .newBuilder().build())
.setUpdateMask(FieldMask .newBuilder().build())
.setValidateOnly(true)
.build();
MetadataFeed response=catalogServiceClient.updateMetadataFeedAsync (request).get();
}
}
}
Node.js
Before trying this sample, follow the Node.js setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Node.js API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Node.js
/**
* This snippet has been automatically generated and should be regarded as a code template only.
* It will require modifications to work.
* It may require correct/in-range values for request initialization.
* TODO(developer): Uncomment these variables before running the sample.
*/
/**
* Required. Update description.
* Only fields specified in `update_mask` are updated.
*/
// const metadataFeed = {}
/**
* Optional. Mask of fields to update.
*/
// const updateMask = {}
/**
* Optional. Only validate the request, but do not perform mutations.
* The default is false.
*/
// const validateOnly = true
// Imports the Dataplex library
const{CatalogServiceClient}=require('@google-cloud/dataplex').v1;
// Instantiates a client
constdataplexClient=newCatalogServiceClient ();
asyncfunctioncallUpdateMetadataFeed(){
// Construct request
constrequest={
metadataFeed,
};
// Run request
const[operation]=awaitdataplexClient.updateMetadataFeed(request);
const[response]=awaitoperation.promise();
console.log(response);
}
callUpdateMetadataFeed();
Python
Before trying this sample, follow the Python setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Python API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Python
# This snippet has been automatically generated and should be regarded as a
# code template only.
# It will require modifications to work:
# - It may require correct/in-range values for request initialization.
# - It may require specifying regional endpoints when creating the service
# client as shown in:
# https://googleapis.dev/python/google-api-core/latest/client_options.html
fromgoogle.cloudimport dataplex_v1
defsample_update_metadata_feed():
# Create a client
client = dataplex_v1 .CatalogServiceClient ()
# Initialize request argument(s)
metadata_feed = dataplex_v1 .MetadataFeed ()
metadata_feed.pubsub_topic = "pubsub_topic_value"
request = dataplex_v1 .UpdateMetadataFeedRequest (
metadata_feed=metadata_feed,
)
# Make the request
operation = client.update_metadata_feed (request=request)
print("Waiting for operation to complete...")
response = operation.result()
# Handle the response
print(response)
REST
To update a metadata change feed, use the
projects.locations.metadataFeeds.patch
method.
Run the following command to update the metadata change feed to remove the entry type filters:
gcurl-XPATCH-d"$(cat<<EOF
{
"filter":{
"entryTypes":[]
}
}
EOF
)""https://${DATAPLEX_API}/metadataFeeds/FEED_ID?updateMask=filter"
Replace the following:
FEED_ID: the metadata change feed ID that you want to update—for example,example-feed
Delete a metadata change feed
You can delete a metadata change feed if you no longer need to receive notifications.
Deleting a metadata change feed stops the feed from publishing new metadata changes to your Pub/Sub topic. However, it doesn't delete the topic or the subscription associated with the feed. You must delete them manually if they're no longer needed.
Before you delete the feed or the subscription, ensure that your subscriber applications have processed all outstanding messages in the Pub/Sub subscription.
C#
Before trying this sample, follow the C# setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog C# API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
C#
usingGoogle.Cloud.Dataplex.V1 ;
usingGoogle.LongRunning ;
usingGoogle.Protobuf.WellKnownTypes ;
publicsealedpartialclassGeneratedCatalogServiceClientSnippets
{
/// <summary>Snippet for DeleteMetadataFeed</summary>
/// <remarks>
/// This snippet has been automatically generated and should be regarded as a code template only.
/// It will require modifications to work:
/// - It may require correct/in-range values for request initialization.
/// - It may require specifying regional endpoints when creating the service client as shown in
/// https://cloud.google.com/dotnet/docs/reference/help/client-configuration#endpoint.
/// </remarks>
publicvoidDeleteMetadataFeedRequestObject()
{
// Create client
CatalogServiceClient catalogServiceClient=CatalogServiceClient .Create ();
// Initialize request argument(s)
DeleteMetadataFeedRequest request=newDeleteMetadataFeedRequest
{
MetadataFeedName=MetadataFeedName .FromProjectLocationMetadataFeed ("[PROJECT]","[LOCATION]","[METADATA_FEED]"),
};
// Make the request
Operation<Empty,OperationMetadata>response=catalogServiceClient.DeleteMetadataFeed (request);
// Poll until the returned long-running operation is complete
Operation<Empty,OperationMetadata>completedResponse=response.PollUntilCompleted();
// Retrieve the operation result
Empty result=completedResponse.Result;
// Or get the name of the operation
stringoperationName=response.Name;
// This name can be stored, then the long-running operation retrieved later by name
Operation<Empty,OperationMetadata>retrievedResponse=catalogServiceClient.PollOnceDeleteMetadataFeed (operationName);
// Check if the retrieved long-running operation has completed
if(retrievedResponse.IsCompleted)
{
// If it has completed, then access the result
Empty retrievedResult=retrievedResponse.Result;
}
}
}
Go
Before trying this sample, follow the Go setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Go API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Go
//go:build examples
packagemain
import(
"context"
dataplex"cloud.google.com/go/dataplex/apiv1"
dataplexpb"cloud.google.com/go/dataplex/apiv1/dataplexpb"
)
funcmain(){
ctx:=context.Background()
// This snippet has been automatically generated and should be regarded as a code template only.
// It will require modifications to work:
// - It may require correct/in-range values for request initialization.
// - It may require specifying regional endpoints when creating the service client as shown in:
// https://pkg.go.dev/cloud.google.com/go#hdr-Client_Options
c,err:=dataplex.NewCatalogClient (ctx)
iferr!=nil{
// TODO: Handle error.
}
deferc.Close()
req:=&dataplexpb.DeleteMetadataFeedRequest{
// TODO: Fill request struct fields.
// See https://pkg.go.dev/cloud.google.com/go/dataplex/apiv1/dataplexpb#DeleteMetadataFeedRequest.
}
op,err:=c.DeleteMetadataFeed(ctx,req)
iferr!=nil{
// TODO: Handle error.
}
err=op.Wait(ctx)
iferr!=nil{
// TODO: Handle error.
}
}
Java
Before trying this sample, follow the Java setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Java API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Java
importcom.google.cloud.dataplex.v1.CatalogServiceClient ;
importcom.google.cloud.dataplex.v1.DeleteMetadataFeedRequest ;
importcom.google.cloud.dataplex.v1.MetadataFeedName ;
importcom.google.protobuf.Empty ;
publicclass SyncDeleteMetadataFeed{
publicstaticvoidmain(String[]args)throwsException{
syncDeleteMetadataFeed();
}
publicstaticvoidsyncDeleteMetadataFeed()throwsException{
// This snippet has been automatically generated and should be regarded as a code template only.
// It will require modifications to work:
// - It may require correct/in-range values for request initialization.
// - It may require specifying regional endpoints when creating the service client as shown in
// https://cloud.google.com/java/docs/setup#configure_endpoints_for_the_client_library
try(CatalogServiceClient catalogServiceClient=CatalogServiceClient .create()){
DeleteMetadataFeedRequest request=
DeleteMetadataFeedRequest .newBuilder()
.setName(MetadataFeedName .of("[PROJECT]","[LOCATION]","[METADATA_FEED]").toString())
.build();
catalogServiceClient.deleteMetadataFeedAsync (request).get();
}
}
}
Node.js
Before trying this sample, follow the Node.js setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Node.js API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Node.js
/**
* This snippet has been automatically generated and should be regarded as a code template only.
* It will require modifications to work.
* It may require correct/in-range values for request initialization.
* TODO(developer): Uncomment these variables before running the sample.
*/
/**
* Required. The resource name of the metadata feed, in the format
* `projects/{project_id_or_number}/locations/{location_id}/MetadataFeeds/{metadata_feed_id}`.
*/
// const name = 'abc123'
// Imports the Dataplex library
const{CatalogServiceClient}=require('@google-cloud/dataplex').v1;
// Instantiates a client
constdataplexClient=newCatalogServiceClient ();
asyncfunctioncallDeleteMetadataFeed(){
// Construct request
constrequest={
name,
};
// Run request
const[operation]=awaitdataplexClient.deleteMetadataFeed(request);
const[response]=awaitoperation.promise();
console.log(response);
}
callDeleteMetadataFeed();
Python
Before trying this sample, follow the Python setup instructions in the
Knowledge Catalog quickstart using
client libraries.
For more information, see the
Knowledge Catalog Python API
reference documentation.
To authenticate to Knowledge Catalog, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
Python
# This snippet has been automatically generated and should be regarded as a
# code template only.
# It will require modifications to work:
# - It may require correct/in-range values for request initialization.
# - It may require specifying regional endpoints when creating the service
# client as shown in:
# https://googleapis.dev/python/google-api-core/latest/client_options.html
fromgoogle.cloudimport dataplex_v1
defsample_delete_metadata_feed():
# Create a client
client = dataplex_v1 .CatalogServiceClient ()
# Initialize request argument(s)
request = dataplex_v1 .DeleteMetadataFeedRequest (
name="name_value",
)
# Make the request
operation = client.delete_metadata_feed (request=request)
print("Waiting for operation to complete...")
response = operation.result()
# Handle the response
print(response)
REST
To delete a metadata change feed, use the
projects.locations.metadataFeeds.delete
method.
Run the following command:
gcurl-XDELETE\
"https://${DATAPLEX_API}/metadataFeeds/FEED_ID"
Replace the following:
FEED_ID: the metadata change feed ID that you want to delete—for example,example-feed
Consume notification messages
After you configure a metadata change feed, Knowledge Catalog publishes messages to the specified Pub/Sub topic when metadata changes occur. To consume these messages, create a Pub/Sub subscription to the topic.
For example, you can create a pull subscription and use the Google Cloud CLI to view messages:
Create a subscription:
gcloudpubsubsubscriptionscreateSUBSCRIPTION_ID--topic=TOPIC_IDReplace the following:
SUBSCRIPTION_ID: the subscription ID that you want to createTOPIC_ID: the Pub/Sub topic ID to which the metadata change feed messages are published.
Pull messages from the subscription:
gcloudpubsubsubscriptionspullSUBSCRIPTION_ID--auto-ack--limit=10Replace the following:
SUBSCRIPTION_ID: the subscription ID from which you want to pull messages
For more information about processing Pub/Sub messages, see Receive messages from a pull subscription. For information about the message format, see Data payload.
What's next
- Read the Metadata change feeds overview.
- Learn about Knowledge Catalog IAM and access control.
- Troubleshoot metadata change feeds.