Chinese character or UTF-8 setting during process the DB connector

[Ruijia]

Dear Support Team,

When i use DB connector to insert some values into MS SQL DB, i found unidentifiable Chinese characters in the DB. Meanwhile the English characters are correctly insert to MS SQL.
The column in MS SQL DB already set to nvarchar type.

Normally when we try to in insert Chinese character into MS SQL, i am following the syntax,
insert into tablename (column1, comuln2) values (N’Vaulue1’, N’Vaulue2’);

But this N seems not working in Tulip connector syntax?
(N’$Applicant$’,N’$Department$’,…

If i only put N, also met error below,

Could you please help me know how to transfer the message via UTF-8 or other format?
Or what else settings i missed?

Thanks in advance!

[gio]

hello @Ruijia - great question, thanks for posting!!

can you try using CONCAT('N', $input$) in the SQL query?? I would try on your behalf, but don’t have MS SQL to test unfortunately.

I’m also eager to hear input from others if someone has come across this.

[Ruijia]

Hi Gio,

Could you please help me to correct if my syntax is not right? Thanks.

[Ruijia]

Hi Gio,

I also tested ‘N’+$input$, not working…

But if if directly put the Chinse characters in the testing, it works…

DB result is as below,

May i know if Tulip has DB connector debug function for messages analysis?

Thanks.

[ermingut]

Same problem here. No special chars inserted here:

Tried to insert: šđčćž ŠĐČĆŽ
Inserted:

As mentioned above “N” syntax works:

[Grant]

Hi @ermingut and @Ruijia,

We’re investigating this on our end and will work to understand what is going on with special characters getting passed to the connector function. I’ll provide an update ASAP on our findings.

Thanks!
Grant

[Ruijia]

Hello @Grant ,
May i know if you have any found out? I heard about Tulip have project in China WuXi, not sure if you have some connection with that project support team and could get suggestion from them?
Thanks

[Grant]

Hi @Ruijia! Yes, we have been able to do some investigation and believe we’ve identified a possible solution here. We’re looking to add explicit support for nvarchar datatypes in the connector - I’ll let you know once I have an update from Engineering on the time frame. Your account manager for these Tulip projects is also working to escalate prioritization with the team as well!

[Ruijia]

Hello Grant,

I had more testing for this case as below,

• The MSSQL accept Chinese characters, the code setting is ‘SQL_Latin1_General_CP1_CI_AS’
• When we want to insert Chinese characters into MS SQL, we need add ‘N’ as the prefix. Such as below,

insert into Tulip_MD_Master values (N’西安杨森Test1’,‘IT’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’);

• But when follows your example, the ‘N’ will be also inert into the DB. This why in below testing result it becomes confusion code.

insert into Tulip_MD_Master values (‘N西安杨森Test2’,‘IT’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’,’’);

• During the Tulip connector transfer the message, seems it add ‘’ for the parameters, but don’t know how it could add N instead of adding ‘N’ in front of the Chinses characters.

Thanks.

[da-noguchi]

Hello @Grant @Ruijia
I’m sorry if my problem has nothing to do with this.
The connector doesn’t work when I enter Chinese characters in the input field.
It works for alphanumeric searches only.
Do you know of any workarounds?

[Grant]

Hi @da-noguchi,

It’s hard to tell if this issue is the same. The first thing I’d mention is that Tulip doesn’t natively support the SELECT * operation as you’ll need to map your selected fields to Outputs in the connector. For example, if your table has the column “work_order”, you might change your query to:

SELECT "work_order" as "Work Order"...

Where “Work Order” is a name of an output parameter that matches the data type of your column.

Can you test with a comparable column and let us know if you’re seeing a specific error message being returned?

Thanks!

[da-noguchi]

Hello @Grant ,

Thank you for your answer.

After “select”, I added what I wanted to output for each.
However, the results will all be “null”.

[Grant]

Hi @da-noguchi,

It looks like the column names are correct and it is finding an existing record. Is it possible that the returned fields that match your test inputs are null for this particular row?

[da-noguchi]

Hello @Grant ,

The output will not all be “null”.
At least the “kiban” part definitely has data in it.

To add to this, there are also alphanumeric codes that can be used to search for the information you want.

[Grant]

Hi @da-noguchi,

Can you send me the results of this same query being run from a SQL client where the results are populated? You can shoot me a message directly with this info as well.