Converting PDF to Base64 - Issues with file sizes

Hi there,

I am currently working on a project with a client where there is a requirement to POST a PDF file, converted to Base64, through a connector.

I am trying to utilise a custom widget created by @Pete_Hartnett (posted in Attachment SendGRid - #17 by Pete_Hartnett) that converts an uploaded PDF file into a Base64 string. The custom widget works perfectly for file sizes of around 500kb or lower - any larger than this however, and an error is encountered where the base64 string "doesn’t match type “string” for prop “b64”

The custom widget converts the PDF into Base64 correctly (I am able to copy and paste the string from the error log report and convert back into a viewable PDF) but it will not set the prop value correctly.

The JSON for the widget can be found below.

customWidget-Base64 Encoder V1.json (3.2 KB)

Is this due to a limit in the allowed length of the string in Tulip? I can see that the full length string is being created in the logs, but it will not populate the b64 prop for some reason.

Any help would be greatly appreciated here!

Many thanks,
Suraj
Bow and Stern IT professional services

Hey @surajp -

This is a bit of a tricky limitation to work around. This is primarily driven by the text type in Tulip having a maximum size limit. These limits are in place to ensure performance for things like queries and aggregations in Tables. With smaller PDFs the base64 encoding is small enough to be used, but bigger PDFs exceed this limit.

We are in the process of allowing customers to configure connector input encoding at the connector level(see details here) and this functionality now exists for all of our image type inputs to connectors. We plan to add support for customer defined encoding for files, but we just haven’t yet gotten here. The hope is to add this functionality in the late summer, but that priority is driven largely by customer demand.

The other approach here would be to change the prop data type from text to imageURL (which does support images encoded as base64, without a size limit), but this is blocked because application/pdf is not a MIME type that can be written to an image url. The other approach is to write to a fileURL prop, but these do not support base64 encoding today. I will ask the team to understand if this is something we could easily support.

Pete

2 Likes

Hi Pete,

Thank you for the speedy response.

A feature to encode files at connector level would be fantastic, is there anywhere I can stay updated on this feature request?

I modified the CW to split the final string into different properties, which allows me to load the entire B64 code into Tulip, albeit as an array with multiple elements (b64_1, b64_2, etc.).
Just trying to pick your brains here - do you think it is possible to send such an array through a POST request, or will we encounter the same issues with maximum size limits as it will still be recognised as a single text field?

Again, appreciate the help and guidance here!

Kind regards,
Suraj