Image input widget can handle Base64String as a data source

Hello all,

We are planning to create an app that can capture a picture of an analog gage and send it to a Web API that returns the value of current needle’s position.

But Image input widget can only handle “image URL” only. On the other hand, Vision Camera widget can handle Base64String as a variable. There is one problem. Vision Camera only supports Windows camera only.

So, my suggestion is to handle Base64String variable with Image input widget. If it works, we can easily send an image to some Web APIs.

Best Regards,
Akira Takahashi

Hey @Akira -

This is a great idea, I ran into this limitation just the other day, you hit on the way I got around this, with vision. I will bring this one to the engineering team to see if we can sneak it in sooner rather than later.

Pete

Just to add in to that. Can tulip handle a long Base64String for input to a variable. More specifically, to a Prop, or embedded in a custom widget?? I am thinking of specifically embedding sounds converted from mp3 to base64.

Thanks,

William Kurth

Hey @wkurth4008 -

Absolutely -

I have a custom widget in the works that populates a report with values passed on props (imagine a tulip app filling out an existing audit form). I passed this base64 out of my Custom Widget as a base64 string. The max string size tables will support is something like 2gb, so absolutely massive

Important to know - text fields have indexes on them in tables that limit their size more significantly to improve query performance, so if you want to store long base64 strings to tables, use image fields in tables.

Pete

Hi
I have a similar use case + one where I need to evaluate the output of a barcode quality tester. It is quite old so I wanted to document the output by capturing a image and pass the result through the Google vision API as Google connector is working flawlessly and now I was looking for a expression component to convert the image stored to base64. I don’t think an expression exist for this conversion.
Could you show me the widget you built, Pete?
Regards
Asger

It makes sense, especially since this function already exists for a photo taken from the vision…

To follow up on my previous post I put together a custom widget which converts an image to base64 (to subsequently allow me to send it through a Google OCR API).

Working with image from the web seems to work fine (both within app and in test mode)

Working with images stored within the instance database appear not to work, and I am unsure if e.g. the custom widget are missing some sort of access to the DB, or if I am missing some javascript code piece :slight_smile:

I hope some of you can advise me on this…

Regards
Asger

Hey @ASharp-J -

When you say database, do you mean tables? if so, base64 must be stored to image fields in tables for a few technical reasons. While the underlying datatype for tables is text (in postgres) so they can support up to 2gb’s of text, the indexes that are used to drastically improve the performance of analytics are a lot more limited on the length of strings that can be stored in table text fields. I also ran into this issue.

I have made a custom widget for a specific usecase that I think can help both of you -

  • The usecase was a situation where a user wanted to populate an existing form with information from a tulip app and store that to a table. Maybe you have a standard audit form that you want Tulip to populate or a standard label you want to generate with a Tulip app
  • Additionally. I wanted this to be flexible to any number of inputs and any text positions, so you could use the same widget for multiple different forms.

This is what I came up with-

  • This widget takes an ImageURL, a web hosted location of the blank for, feel free to use “https://i.ibb.co/XLp9RrW/kaizen-audit.jpg” as a working example, but this could be any publicly accessible image file.
  • It also takes a font size (self explanatory) and a list of text, x and y positions. When the save() function is called the canvas is encoded as base64 and passed back as an event. From here it can be stored to a table.

Fair Warning - This widget really was a proof of concept for me, its far from perfect, and hasnt gone through Tulip’s standard code review process. Use it at your own risk.
customWidget-Form V2.json (1.9 KB)

I have also used custom widgets with prop text inputs that were base64 to then pass them to API endpoints for stuff like anomoly detection. I can’t share that full widget because it includes some API keys, but here is the meaningful JS:

getValue('imageBase64String', () => {detectAnomalies();});

function b64toblob(image) {
    const byteCharacters = window.atob(image);
    const byteNumbers = new Array(byteCharacters.length);
    for (let i = 0; i < byteCharacters.length; i++) {
      byteNumbers[i] = byteCharacters.charCodeAt(i);
    }
    const byteArray = new Uint8Array(byteNumbers);
    const blob = new Blob([byteArray], {type: 'application/octet-stream'});
    return blob;
}
function getPred(preds){
    let pre =  preds[0];
    console.log(pre.probability)
    if(pre.probability > .7){
        fireEvent('Color',pre.tagName)
    }
    else{
        fireEvent('Color','No Match ' + pre.probability)
    }
}

async function detectAnomalies() {

    let image= getValue("imageBase64String");
    const url = 'https://eastus.api.cognitive.microsoft.com/customvision/.....';
	const img = b64toblob(image);
    $.ajax({
        url: url,
        type: 'post',
        data: img,
        cache:false,
        processData: false,
        headers: {
            'Prediction-Key': 'e6abf....',
            'Content-Type': 'application/octet-stream'
        },
        success: (response)=>{
            console.log(response["predictions"]);
            getPred( response["predictions"]);
        },
        error: (err)=>{
            console.log(err);
        },
        async: false,
    });
    
}

Hope this helps-
Pete