Proposal
Based on the need to scrape embedded schema.org structured data from LinkedIn post pages (as described here) and potential future use cases involving schema.org I propose implementing new /api/3/get/ proxy route. This new /api/3/get/schemaorg proxy route would work much like /api/3/get/activitypub:
- it would accept
urlquery param with the target URL containing embedded schema.org structured data - it would accept
typequery param containing valid schema.org type - it would return the first occurrence of scrapped schema.org JSON data matching required
typewithContent-Typeset toapplication/ld+json
The scraper needs to:
- find all
script[type="application/ld+json"]elements and extract their JSON payload - filter out those who's
$["@context"] !== "http://schema.org" - return the first payload whose type matches the requested type:
$["@type"] === query.type
Using @martin.sweeney 's example post from keyoxide/doipjs#18, the request would look like this:
GET https://keyoxide.org/api/3/get/schemaorg?type=SocialMediaPosting&url=https://www.linkedin.com/posts/martinsweeny_verifying-my-openpgp-key-openpgp4fpr08a83f21244c4876d9ec2c6a5f88f7e8f01d4198-activity-6884620821181063168-WsxX
and it would return:
Content-Type: application/ld+json
{
"@context": "http://schema.org",
"@type": "SocialMediaPosting",
"@id": "https://www.linkedin.com/posts/martinsweeny_verifying-my-openpgp-key-openpgp4fpr08a83f21244c4876d9ec2c6a5f88f7e8f01d4198-activity-6884620821181063168-WsxX",
"author": {
"name": "Martin â–1⁄2 Sweeny",
"image": {
"url": "https://media.licdn.com/dms/image/D4E03AQEPi12R3WJiLg/profile-displayphoto-shrink_200_200/0/1693155735016?e=2147483647&v=beta&t=F9Zmr1SdpAZJ-tvqRRCzIk7Gnugofrj48Y6c-NUGw2w",
"@type": "ImageObject"
},
"url": "https://ca.linkedin.com/in/martinsweeny",
"@type": "Person"
},
"datePublished": "2022年01月05日T22:25:13.826Z",
"articleBody": "[Verifying my OpenPGP key: openpgp4fpr:08a83f21244c4876d9ec2c6a5f88f7e8f01d4198]",
"hasPart": {
"@type": "WebPageElement",
"isAccessibleForFree": false,
"cssSelector": ".details"
}
}
### Proposal
Based on the need to scrape embedded schema.org structured data from LinkedIn post pages (as described [here](https://codeberg.org/keyoxide/doipjs/pulls/18#issuecomment-1637237)) and potential future use cases involving schema.org I propose implementing new [`/api/3/get/` proxy route](https://codeberg.org/keyoxide/keyoxide-web/src/commit/b8c94ebc0b/src/api/v3/proxy_get.js). This new `/api/3/get/schemaorg` proxy route would work much like [`/api/3/get/activitypub`](https://spec.keyoxide.org/spec/2/#:~:text=Returns%3A%20JSON%20object-,%2Fapi%2F3%2Fget%2Factivitypub,Returns%3A%20JSON%20object,-4.%20Data%20schemas):
- it would accept `url` query param with the target URL containing embedded schema.org structured data
- it would accept `type` query param containing valid [schema.org type](https://schema.org/docs/full.html)
- it would return the first occurrence of scrapped schema.org JSON data matching required `type` with `Content-Type` set to `application/ld+json`
The scraper needs to:
1. find all `script[type="application/ld+json"]` elements and extract their JSON payload
2. filter out those who's `$["@context"] !== "http://schema.org"`
3. return the first payload whose type matches the requested type: `$["@type"] === query.type`
Using @martin.sweeney 's example post from https://codeberg.org/keyoxide/doipjs/pulls/18, the request would look like this:
```
GET https://keyoxide.org/api/3/get/schemaorg?type=SocialMediaPosting&url=https://www.linkedin.com/posts/martinsweeny_verifying-my-openpgp-key-openpgp4fpr08a83f21244c4876d9ec2c6a5f88f7e8f01d4198-activity-6884620821181063168-WsxX
```
and it would return:
```
Content-Type: application/ld+json
{
"@context": "http://schema.org",
"@type": "SocialMediaPosting",
"@id": "https://www.linkedin.com/posts/martinsweeny_verifying-my-openpgp-key-openpgp4fpr08a83f21244c4876d9ec2c6a5f88f7e8f01d4198-activity-6884620821181063168-WsxX",
"author": {
"name": "Martin â–1⁄2 Sweeny",
"image": {
"url": "https://media.licdn.com/dms/image/D4E03AQEPi12R3WJiLg/profile-displayphoto-shrink_200_200/0/1693155735016?e=2147483647&v=beta&t=F9Zmr1SdpAZJ-tvqRRCzIk7Gnugofrj48Y6c-NUGw2w",
"@type": "ImageObject"
},
"url": "https://ca.linkedin.com/in/martinsweeny",
"@type": "Person"
},
"datePublished": "2022年01月05日T22:25:13.826Z",
"articleBody": "[Verifying my OpenPGP key: openpgp4fpr:08a83f21244c4876d9ec2c6a5f88f7e8f01d4198]",
"hasPart": {
"@type": "WebPageElement",
"isAccessibleForFree": false,
"cssSelector": ".details"
}
}
```