Thanks Aditya for the detailed proposal. I appreciate the thorough overview and possible implementation methods that you've presented.

Just to ensure we're all on the same page, let's define the scope of our discussion. We're primarily focusing on Augmented Reality applications that leverage the camera on an Android mobile device. The goal is to overlay 2D or 3D content over a specific image in the real world, as seen in augmented picture books such as the one pictured in the GIF below. Is my understanding correct?

book-7
Image: Adrien Mondot and Claire Bardainne

To clarify, are you considering the addition of a built-in library of 3D models to Processing for Android? From what I understand, based on Gaurav's Medium article, the main concern seems to be around how to handle the image marker database given ARCore's limitation of 1000 images per database. To confirm, is this limitation specific to each application?

I believe that Processing creators will want to load their own 3D models (or build geometry at runtime in OpenGL), and use their own image markers. This means that Processing's AR support should focus on facilitating the process of adding user's custom models and images, rather than providing a built-in database of models or images. Therefore the "Pure Dynamic way" seems to best align with the open-ended nature of Processing. We can definitely include a few smaller copyright-free 3D models in the examples for the library, but our main focus should be providing the tools necessary for users to incorporate their own work.

Let me know if I'm correctly understanding the scope of the discussion or if there's anything else that needs further clarification.

Best,
Raphaël

PS: While the idea of loading models from an online database, like Firebase, is intriguing, I think it falls a bit outside of the project's scope. That is, unless ARCore has some specific quirks that require a distinct method for loading files from a remote server.

I noticed the inclusion of the article "The Ultimate Guide to Markerless Augmented Reality" in your references. Since this piece is about markerless AR, it seems to fall outside the scope of our current discussion on image markers. Could you clarify this for me? Was this link intended?

However, this made me curious about other aspects of the ARCore API. Are you also considering incorporating features like plane detection, depth-based occlusion or face augmentation in this project?

Hi @ranaaditya and @p4puniya,

I appreciate your ideas for the AR project, but I worry that a focus on the remote hosting of assets adds complexity that could make things more challenging than necessary for our user base.

In my opinion, the main focus should be on simplifying the process of loading user's custom models and image markers from the sketch folder (locally). The 1000 image-cap should be more than enough for most use cases.

I believe that an online database of image markers is superfluous and problematic. Artists will likely prefer to use their unique images, and having to download markers or textures could pose problems for those with limited bandwidth or in offline situations. Additionally, an online database would require consistent hosting and maintenance, which presents yet another operational concern.

I'm interested in hearing what Andres (@codeanticode) Colubri thinks about this.

Best,
Raphaël

Hello everyone, sorry for the delay in responding to this thread!

Very interesting points and references by @ranaaditya and @p4puniya, thank you guys.

After reading the Google's developer documentation on augmented images, and considering the discussion so far, my own opinion would be to simply use the local database functionality in ARCore, following the purely dynamic approach, where users would create the image database on the fly from the sketch by adding PImage objects to it.

Also, I think we should minimize the need of any new API, if my understanding of ARCore's AugmentedImage class is correct, we would be able to integrate image trackables into the current infrastructure of the AR library by adding a new trackable type alongside wall and floor, i.e.: image, so we can reuse the existing trackable functionality, so a sketch using image markers could look like this:

import processing.ar.*;
ARTracker tracker;
void setup() {
 fullScreen(AR);
 tracker = new ARTracker(this);
 PImage img = loadImage("example.png");
 tracker.addImage(img);
 tracker.start();
}
void draw() {
 for (int i = 0; i < tracker.count(); i++) {
 ARTrackable t = tracker.get(i);
 println("Trackable", i, t, t.isImage());
 } 
}

A cloud database would be cool, but as @SableRaf pointed out, this would introduce significant complications and extra maintenance. And advanced users should still be a able to implement their own remote marker databases using the Android API directly.

Hope that my comments are useful :-)

I was thinking how to Add the Images in the AugmentedImageDatabase for ARMarkers, and there can be 2 ways:

1. Adding them one by one and writing boiler plate code for each image in PDE to add them to the AugmentedImageDatabase.
2. Making a database file using arcoreimg tool and adding that to the augmented image database.
   I personally prefer the second method, as it will be easier to index all the images. The only short coming being that the user needs to go through a few extra steps to make the database file. So, should I use the first method or the second to implement this?
   P.s. If we use the second method, once we are done with our project, we can make some tutorials for the user to make the image database using the CLI tool and add it to the project.

Hey @p4puniya

Just caught up with your discussion with Andres, really interesting stuff! It's clear you're taking the user experience to heart when thinking about AR markers, which is just great.

Your conversation reminded me of a key aspect of Processing's design philosophy. The main idea is to make it easy to do simple things, but to also allow a more experienced programmer to do complicated things within the same language.

Sometimes, what seems like a great solution from an engineering perspective (think using a CLI tool, or loading assets from a CDN) can actually be daunting for a beginner. This is a tricky balance to achieve and there's no one-size-fits-all solution to make your designs beginner-friendly. However, keeping this goal in mind can help guide you in future design decisions for the project.

For further insights into what makes Processing unique, I recommend reading A Modern Prometheus by Ben Fry and Casey Reas (the creators of Processing).

Hi @codeanticode, I'm making the ARtrackables.isImage() method. I just wanted to discuss what would be it's exact use. I read that it will work to use both Image and planes. So should I make it as a boolean method which will return false in case we have a plane and true for Images?
I saw that in ARGraphics, there are specific IDs for each plane:

 static protected final int PLANE_FLOOR = 0;
 static protected final int PLANE_CEILING = 1;
 static protected final int PLANE_WALL = 2;
 static protected final int POINT = 3;

And then these IDs are checked by specific functions in ARTrackable like this:

 public boolean isCeilingPlane() {
 int idx = g.trackableIndex(id);
 return g.trackableType(idx) == ARGraphics.PLANE_CEILING;
 }

So, following the above example, I can add a new Index for images in ARGraphics:
static protected final int IMAGE = 4;
And make isImage() method in ARTrackable() like:

public boolean isImage(){
 int idx = g.trackableIndex(id);
 return g.trackableType(idx)== ARGraphics.IMAGE;
 }

Am I correct? Also, are there any other changes that I need to make for tracking the Image?

Hi @codeanticode,
I've been facing an issue when running the addImage functionality:

System.err: An error occurred in ARCORE: Attempt to read from field 'long com.google.ar.core.Session.nativeWrapperHandle' 
on a null object reference in method 'void com.google.ar.core.Config.<init>(com.google.ar.core.Session)'

This is my current code:

public void addImage(String name, PImage img) {
 Bitmap bitmap = (Bitmap)img.getNative();
 int imgIndex = imgDB.addImage(name,bitmap); 
 Config config= new Config(session);
 
 config.setAugmentedImageDatabase(imgDB);
 session.configure(config);
 }

I tried the following to resolve this but it doesn't seem to work:

1. Using the default Session & Context of ARCore and initialising it to the config as done in the default AR Core apps. (error not resolved)
2. Initialising session using session= new Session(session) before passing it to the context. (I tried passing arg.surfar.session as well but still the same error.)
3. I saw that in arg.surfar.session was passed as a parameter instead of session when initialising the imgDB in ARTracker.
   Surfar is a ARSurface object (defined in ARGraphics) and we are passing it's session when defining imgDB. So I tried to do the same. Using arg.surfar.session , but still no luck.

I'm out of ideas here and can't seem to find a solution for this. Please help me resolve this issue.

I reviewed Gaurav's PR and made a number of changes to the augmented images work alongside the existing plane trackables. Everything is merged into main now. I did not tests extensively, but seems to work well, the photos below are from the Images example I added to the AR library:

image

@p4puniya @ranaaditya can you guys test some more to make sure all other functions in the AR library work as expected? I refactored the code a bit and I think everything should be the same but you never know :-)

Closing the discussion on this topic since all latest changes are merged into the main branch :-)