CloudScript (Legacy): http GET request returns invalid file data

Question

question

petergrimsehl asked Aug 18, '23

CloudScript (Legacy): http GET request returns invalid file data

Hello,

I'm using a legacy cloud script where I access a file stored in the title player player account by making a GET request to the DownloadUrl provided in the meta data retrieved by GetFiles(). The file contains binary data that was uploaded by the client using the File / Data REST API and libcurl to put the contents. This file should be uploaded to another server to be attached to an open issue.

I verified that the uploaded file contents are valid by manually downloading the file from the players file section.

Here is a part of the script up to the GET request.

 // ...
 var entityProfile = context.currentEntity;
    
 var getFileRequest = {
     Entity : entityProfile.Entity
 };
    
 var file_url;
 var result = entity.GetFiles( getFileRequest );
    
 if ( result && result.Metadata )
 {
     var file_meta = result.Metadata["bug.t7s"];
    
     if ( file_meta )
     {
         file_url = file_meta.DownloadUrl;
     }
    
     if ( file_url )
     {
         var headers = {
             "Accept": "application/octet-stream"
         };
    
         var get_response = http.request( file_url, "get", "", "", headers, true );    
 // ...

After executing the this GET request, get_response.length contains the correct number of characters (== file size), but debug printing the individual characters of the string get_response showed that several characters of the original byte-sequence where replaced by the unicode replacement character (0xFFFD), which renders the received file contents invalid for further processing.

The final step would be a REST call to an external service, using get_response as content of a multipart/form-data post request

 // Create the attachment.
 var auth = "omitted auth string";
    
 var headers = {
     "Authorization": auth,
     "X-Atlassian-Token": "no-check"
 };
    
 content =
 "--partboundary" +
 "\r\nContent-Disposition: form-data; name=\"file\"; filename=\"bug.t7s\"" +
 "\r\nContent-Type: application/octet-stream" +
 "\r\n" +
 "\r\n" + get_response +
 "\r\n--partboundary--"
    
 var url = "omitted target url";
 var httpMethod = "post";
 var contentType = "multipart/form-data; boundary=partboundary";
    
 var response = http.request(url, httpMethod, content, contentType, headers, true);

This calls succeeds but posts invalid / broken data to the target site.

Have you any advice how I can solve this problem?

CloudScript

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 1 · 2023-09-05T10:00:35Z

Neils Shi answered Sep 5, '23

Could you tell us why you want to get the entity files via CloudScript? Since some characters will be replaced by Unicode replacement characters in CloudScript, you may call the API GetFiles on client side. And the CloudScript is a legacy feature, we suggest you use the CloudScript using Azure Functions.

1

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

petergrimsehl commented · Sep 12, 2023 at 12:06 PM

The clients should not be able to upload data to the external service.

But since it seems that the CloudScript API is no longer maintained, this issue can be closed.

0 ·

Answer 2 · 2023-08-21T10:08:07Z

Neils Shi answered Aug 21, '23

When you convert binary data to string in JavaScript, the Unicode replacement character (0xFFFD) is used to replace an incoming character whose value is unknown or ill-formed in a Unicode Transformation Format (UTF) conversion. Could you tell us more details about how you debug print the individual characters of the string “get_response?

6

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

petergrimsehl commented · Aug 22, 2023 at 01:59 AM

Hi,

I'm using the follwing function for debug output

     function stringToBytes ( str ) {
    
         log.info( "Converting string of length: " + str.length );
    
         var ch, st, re = [];
         for (var i = 0; i &lt; str.length; i++ ) {
           ch = str.charCodeAt(i);  // get char 
           st = [];                 // set up "stack"
           do {
             st.push( ( ch &amp; 0xFF ).toString(16).padStart(2, "0").toUpperCase() );  // push byte to stack
             ch = ch &gt;&gt; 8;          // shift value down by 1 byte
           }
           while ( ch );
    
           //log.info( "Stack has number of elements: " + st.length );
    
           // add stack contents to result
           // done because chars have "wrong" endianness
           re = re.concat( st.reverse() );
         }
         // return an array of bytes
         log.info( "Returning array of length: " + re.length );
    
         return re;
     }

0 ·

Neils Shi petergrimsehl commented · Aug 25, 2023 at 10:06 AM

I cannot reproduce your issue. After I uploaded the binary data, I can get the correct response without unicode replacement character (0xFFFD), you may share us with your uploaded binary data so that we can do some research.

0 ·

petergrimsehl Neils Shi commented · Aug 29, 2023 at 02:22 AM

I uploaded a test file containing 256 bytes (0-255 ascending).

My debug print function outputs an array of 512 bytes for get_response after the GET call:

 00,01,02,03,04,05,06,07,08,09,0A,0B,0C,0D,0E,0F,
 10,11,12,13,14,15,16,17,18,19,1A,1B,1C,1D,1E,1F,
 20,21,22,23,24,25,26,27,28,29,2A,2B,2C,2D,2E,2F,
 30,31,32,33,34,35,36,37,38,39,3A,3B,3C,3D,3E,3F,
 40,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,
 50,51,52,53,54,55,56,57,58,59,5A,5B,5C,5D,5E,5F,
 60,61,62,63,64,65,66,67,68,69,6A,6B,6C,6D,6E,6F,
 70,71,72,73,74,75,76,77,78,79,7A,7B,7C,7D,7E,7F,
 FF,FD,FF,FD,FF,FD,FF,FD,FF,FD,FF,FD,FF,FD,FF,FD,
 (... for the rest of the file, ommited due to character limit in replies.)

So apparently all byte values >= 0x80 are broken.

Downloading directly from the players file section gives a file with the correct contents.

Downloading the file as attachment after uploading it to the external server with multipart/formdata gives a file of size 512 bytes with broken data after byte value 0x7F, mostly replaced with 0xBDEF.

0 ·

Show more comments

question