Google App Engine

Checking for authorizations

GAE javascript actions are stateless web services, that is to say, they do not store conversational state among different calls.

These actions can be invoked directly or, even better, through an alias (api?cmd=...).

In any case, an authentication process is required, in order to successfully invoke an action.

Authentication can be performed in 2 alternative ways:

  • by invoking "login" standard ws first, get a token and use it for any subsequent invocation of a GAE action by pass the token along with the other input parameters

  • by invoking directly a GAE action an pass credentials along with the other input parameters

In any case, credentials are required and must be provided always as request headers.

Moreover, in the first case the needTmpToken is required too, in order to ask Platform for GAE to get back a token to use in the next requests.

Optionally, it is possible to ask Platform to fetch authorizations as well, when carrying out the authentication process. Authorizations are the list of granted roles associated to the specified user.

This can be carried out b y including in the request the "loadRoles=Y" header parameter.

Finally, the utils.checkRoleId(roleId) method can be included in any GAE action, in order to force Platform for GAE to check for that role as a role bound to the current user. This checking is performed ONLY IF "loadRoles=Y" has been previously included in the authentication request.

Reading data from Datastore

When there is the need to read a sigle entity, it is strongly recommended to use the following method to read a single record by key instead of executing a query where the filter condition limits the number of record, since reading a single record by key is cheaper.

Syntax

var json = utils.getEntityAsJSON(entityName, key, maxCachedEntities);
utils.setReturnValue(json);

Parameters

Reading data from Datastore

When there is the need to read a sigle entity, it is strongly recommended to use the following method to read a single record by key instead of executing a query where the filter condition limits the number of record, since reading a single record by key is cheaper.

Syntax

var json = utils.getEntityAsJSON(entityName, key, maxCachedEntities, expirationTime);
utils.setReturnValue(json);

Parameters

Execute a long HTTP(s) requestExecute a long HTTP(s) request

AppEngine is a high scalable web container, used to run web services.

In order to provide scalability, it limits all HTTP requests to 30 seconds only. It goes without saying that in case of longer operations, like in case of a sequence of writing operations, there is the need for a queue and the real operations must be postponed to the queue execution.

The best approach is to enqueue operations in a GAE action, using something like:

var uuid = utils.enqueueActionWithNoteAsString(
      "QueueName",
      actionIdInvokedByTheQueue,
      { /* parameters to pass forward to the enqueued action*/ },
      null,
      null,
      false,
      null
);

This method gets back a unique identifier for the element in queue.

At this point, there are two possible scenarios:

  • the client invoking the current action does not need a feedback: that's all

  • the client needs a feedback and this action must wait for the termination of the enqueued action

In the latter case, since the HTTP request launched by the client will terminate in 30 seconds (because GAE will force its termination), we can wait after the enqueueActionWithNoteAsString command, but not for more than 30 seconds. It can be helpful this method:

var json = utils.blockingWaitAllElements(20,[uuid]);
utils.setReturnValue(json);

In this example, this method will wait up to 20 seconds, until all enqueued elements specified in list will be terminated.

If after the specified timeout (expressed in seconds), not all of the elements are still finished, the method ends gracefully and get back the list of not terminated element ids, which can be provided to the client.

Consequently, the client should invoked a second request, to a third action which should contain only the previous instructions, so that it can determine when the actions are terminated.

Capturing enqueued action outcome

The enqueued action can also provide a feedback to the invoking action or to any other action; the action return value (JSON content) is automatically cached internally, referred by the enqueued action uuid. That means that if the enqueued action terminates by providing some content (either a successful or a wrong outcome), like in this scenario:

var response = { success: true, someData: "..." };
if (somethingWrong) {
 response = { success: false, message: "..." };
}
utils.setReturnValue(JSON.stringify(response));

this content can be accessed, for example from the calling action, through the uuid:

var uuid = utils.enqueueActionWithNoteAsString(
      "QueueName",
      actionIdInvokedByTheQueue,
      { /* parameters to pass forward to the enqueued action*/ },
      null, // priority: not managed
      null, // process wait time: not managed
      false, // log execution: not managed
      null, // optional payload, expressed as a String
      null  // optional: max retries as a number
);
var json = utils.blockingWaitAllElements(20,[uuid]);
if (json=="[]") {
  // enqueued action has terminated on time...
  var json = utils.getValueInCache(uuid); // this value automatically expires in 1 minute
  var response = JSON.parse(json); // { success: ..., ...}
}
utils.setReturnValue(json);

Bear in mind that cached response for an enqueued action is available up to 1 minute, after that time, the content is automatically removed from the cache.

Checking for element termination

When enqueuing an element in AppEngine, the "enqueueActionWithNoteAsString" method returns an uuid identifying the specific element appended to the queue.

Optionally, you can pass forward any payload (for example a unique identifier as String or a JSON String) to such method, using the "note" argument. You can later use this information to check for the element termination, using the "getElementFromQueueByNote" method, which enquiry the queue management system, searching from an element having the specified payload.

Syntax

var json = utils.getElementFromQueueByNote(String payload,String namespace);

The payload argument is used to search for an element still inside the queue, having that payload bounded.

If it exists, the method returns the element in queue (a JSON String representing the enqueued element).

An example of returned JSON is:

{
  "appId": "GAE",
  "companyId": "00000",
  "siteId": 100,
  "queueName": "queue1",
  "actionId": 119,
  "note": "123",
  "status": "ERROR",
  "createDate": "Jun 3, 2021 3:59:53 PM",
  "json": "{...}"
}

There are two possibilities for having still and element returned:

  • the element is still in queue; in such a case, the "status" attribute is set to "ENQUEUED"

  • the element has been dequeued but not processed correctly and error was fired during the elaboration; in such a case, the "status" attribute is set to "ERROR"

If the method returns null, it means the element has been already processed and removed from the queue.

Example

Thread where the element is enqueued:

var myDocumentId = "...";
var documentToProcess = {
  id: myDocumentId,
  ...
};

var uuid = utils.enqueueActionWithNoteAsString(
      "QueueName",
      actionIdInvokedByTheQueue,
      documentToProcess, // parameters to pass forward to the enqueued action
      null, // priority: not managed
      null, // process wait time: not managed
      false, // log execution: not managed
      myDocumentId, // payload, expressed as a String
      null  // optional: max retries as a number
);




Another thread, checking for element termination

var myDocumentId = "..."; // the same value used in the firdt thread
var namespace = "...";

var element = utils.getElementFromQueueByNote(myDocumentId, namespace);
if (element!=null) {
    // the enqueued element can be in queue or dequeued and terminated with errors
    var outcome = JSON.parse(element);
    if (outcome.status=="ENQUEUED") {
      // element still enqueued...
    } 
    else {
      // what to do in such a case?
    }
}
else {
    // the element has been dequeued and processed correctly and no more in queue
}

Concurrent writing operations

As long as the updateObject/updateObjects are used, the Google Datastore automatically provides a checking which avoids the possibility to update the same object multiple times, starting from old states of the object: it means that if concurrent requests to writing operations are performed, only the ones starting from a consistent state from the Datastore will be allowed. The others will fire an exception, which must be captured and managed appropriately by the GAE javascript code.

Let's take the classical example of a counter table "Counters" whose purpose is to reckon the next value for a counter. A Datastore object used for that goal would be something like:

The right GAE javascript action should manage the possibility of a concurrent access exception and attempt multiple times to get the next value:

var todayStr = new Date().toISOString().substring(0, 10);
var sectional = reqParams.ente+"_"+reqParams.filiale+"_"+todayStr+"_"+reqParams.registerNr;
var key = userInfo.companyId+"_"+userInfo.siteId+"_"+reqParams.docType+"_"+new Date().getFullYear()+"_"+sectional;
var attempts = 0;
var currentValue = null;
var json = utils.getEntityAsJSON("Counters",key, 0); // always set to 0 the cache, so that the record is always read

if (json==null) {
  // insert record...
  currentValue = 1;
  try {
    // ...
  }
  catch(e) {
    // insert failed, because in the meantime another concurrent request has inserted the same record:
    // let's convert the insert to update, i.e. let's go on
    json = utils.getEntityAsJSON("Counters", key, 0); // always set to 0 the cache, so that the record is aways read
  }
}

if (json!=null) {
  // update record concurrently
  do {
    try {
      attempts++;
      vo = JSON.parse(json);
      vo.currentValue = vo.currentValue+1;
      utils.updateObject(vo,...);
      currentValue = vo.currentValue;
      break;
    }
    catch(e) {
      // concurrent exception: reload a freshed version of the record
      json = utils.getEntityAsJSON("Counters", key, 0); // always set to 0 the cache, so that the record is aways read
    }
  }
  while(attempts<100);
  if (currentValue==null)
    throw "Update not performed: try later";
}

utils.setReturnValue(JSON.stringify({ success: true, currentValue: currentValue }));

How to manage distributed transactions and SAGA

Since Google Datastore is a NoSQL database, the ACID (atomic, concurrent, isolated, durable) properties supported by a relational database cannot be applied.

Consequently, when multiple writing operations on different entity types are required, it is not possible to count on the Database behavior. The best way to manage a distributed transaction over multiple entity types, is to manage it as for a micro-service architecture, based on the SAGA pattern (see for example: https://blog.couchbase.com/saga-pattern-implement-business-transactions-using-microservices-part-2\, using the "command-orchestration sequencing logic", where a main javascript action would orchestrate all operations, by invoking each of them sequentially.

In case of an error fired by any of the invoked actions, the main javascript action has the duty to "undo" any writing operation already carried out, by deleting them. That means that every invoking object should not only provide a "writing operation" but also an "undo" operation, to clean up any writing operation already done.

When comparing Platform with the schema above:

  • the "Order Service" represents the GAE javascript action invoked by a web/mobile client

  • the "Order Saga Orchestrator" is the GAE javascript action enqueued by the previous one, which is invoked asynchronously

  • "Payment Service", "Stock Service" and "Delivery Service" represent additional GAE javascript actions which can be called by the previous enqueued action. In the easiest case, we can embed this logic as a sequence of writing operations within the "Order Saga Orchestrator" action, which has the purpose of managing the "rollback" operations as well, in case of errors during the sequence of writing operations. In the most complex scenario, these 3 additional actions are enqueued too by the "Order Saga Orchestrator" into:

    • 3 distinct queues, if it makes sense to parallelize these 3 operations; a blockingWaitAllElements is required, in order to check out if all operations completed correctly or to manage the undo task

    • a unique queue, where the "Payment Service" is enqueued, waiting for its termination through the blockingWaitAllElements method; when completed correctly, the second actions is enqueued; same approach for the third. Undo operations are managed after each blockingWaitAllElements termination, if needed.

How to recycle enqueued elements with problems

Every time a GAE action enqueues an element to Google Cloud Task, the same element is also saved in the QueueElements entity by Platform. This entity contains:

  • id - element identifier

  • json - the input data to pass forward to the element to process

  • app id, company id, site id, namespace

  • status

The status attribute is set to ENQUEUED every time and element is enqueued.

When the Task manager extracts the element, it is processed and a few alternative outcomes can arise:

  • the elaboration terminates correctly: the record is removed automatically from QueueElements entity

  • the elaboration was interrupted by an error:

    • if the max number of retries has not reached yet, the record is marked with ERROR status and managed again, up to the max number of retries set

    • f the max number of retries has been reached, the record is marked with ERROR status: not more attempts and carried out and the record remains in QueueElements for ever

Another abnormal scenario is when an element is never extracted from the Task manager: in such a case, the record still remains in QueueElements with status ENQUEUED.

It is possible to use the following GAE javascript function to re-process a single element in QueueElements: after re-enqueuing it, the record is also removed from QueueElements.

var rows = utils.reinsertElements(
    null, //String companyId,
    null, //Long actionId,
    null, // String queueName,
    null, //String status,
    "019d9ca7-5c06-464a-8eb7-00afb8cbe47e" // id in QueueElements
);
utils.debug("reinsertElements: "+rows);

It is possible to use the same GAE javascript function to re-process multiple elements in QueueElements: the ones matching the filtering conditions passed forward to the method; after re-enqueuing a record matching the filtering criteria, that record is also removed from QueueElements.

var actionId = ...
var rows = utils.reinsertElements(
    "00000", // reinsert only records having this company id
    actionId, //reinsert only records having this action id,
    "MYQUEUE", // reinsert only records having this queue name,
    "ERROR", // reinsert only records having this status,
    null // do not set it
);
utils.debug("reinsertElements: "+rows);

Execute an HTTP(s) connection using NLTM authentication

result expressed as a String (e.g. a JSON or XML result content)

Syntax

var json = utils.getWebContentWithNTLM(
  uri, 
  contentType, 
  httpMethod, 
  requestBody, 
  user, 
  pwd,
  workstation,
  domain,
  settings
);

Details

 Returns the HTTP response, expressed as a String (e.g. a JSON or XML result).

HTTP response codes included between 200 and 399 are managed as correct answers and the response is sent back through the "json" return variable.

In case of HTTP response codes above or equal to 400, an exception is fired an the exception content would contain the message sent back by the invoked web service; consequently, it would be better to surround this instruction between try-catch.

The "settings" js object can include the following attributes:

  • "connectionTimeout: an optional number defining the timeout for the connection

  • "headers": a js object containing request headers, e.g. required credentials

Example:

var requestBody = "...";
var json = utils.getWebContentWithNTLM(
    'https://...', 
    "application/json; charset=utf-8",
    "POST",
    requestBody,
    "username",
    "pwd",
    "workstation",
    "domain",
    null
); 

Call a business component from a server-side js business component

It is available a js function to invoke from within a server-side js b.c. (not from an action) to call another business component.

Basically, it can be used to pass forward all request parameters received in input from the starting b.c and get a response to use as the final result to pass back.

Syntax

callBusinessComponent(
  Long compId,
  Map additionalReqParams,
  Map decodedReqParams,
  Map decodedFilterNames
)

Example of a server-side js b.c. invoking a b.c. for grids

var json = utils.callBusinessComponent(109,{},{},{
     codice: "uid", // replace "codice" input req param with "uid"
     descrizione: "companyTitle"
});
var res = JSON.parse(json);
// scroll all results returned by the called b.c. and create a new 
// response, containing the data needed
var list = [];
for(var i=0;i<res.valueObjectList.length;i++) {
    list.push({
        codice: res.valueObjectList[i].uid,
        descrizione: res.valueObjectList[i].companyTitle
    });
}

var _res = {
    valueObjectList: list,
    moreRows: res.moreRows
};
utils.setReturnValue(JSON.stringify(_res));

Execute a long HTTP(s) requestExecute a long HTTP(s) request

AppEngine is a high scalable web container, used to run web services.

In order to provide scalability, it limits all HTTP requests to 30 seconds only. It goes without saying that in case of longer operations, like in case of a sequence of writing operations, there is the need for a queue and the real operations must be postponed to the queue execution.

The best approach is to enqueue operations in a GAE action, using something like:

var uuid = utils.enqueueActionWithNoteAsString(
      "QueueName",
      actionIdInvokedByTheQueue,
      { /* parameters to pass forward to the enqueued action*/ },
      null,
      null,
      false,
      null
);

This method gets back a unique identifier for the element in queue.

At this point, there are two possible scenarios:

  • the client invoking the current action does not need a feedback: that's all

  • the client needs a feedback and this action must wait for the termination of the enqueued action

In the latter case, since the HTTP request launched by the client will terminate in 30 seconds (because GAE will force its termination), we can wait after the enqueueActionWithNoteAsString command, but not for more than 30 seconds. It can be helpful this method:

var json = utils.blockingWaitAllElements(20,[uuid]);
utils.setReturnValue(json);

In this example, this method will wait up to 20 seconds, until all enqueued elements specified in list will be terminated.

If after the specified timeout (expressed in seconds), not all of the elements are still finished, the method ends gracefully and get back the list of not terminated element ids, which can be provided to the client.

Consequently, the client should invoked a second request, to a third action which should contain only the previous instructions, so that it can determine when the actions are terminated.

Get directory path

Syntax:

var path = utils.getDirectoryPath(directoryId)

Caching values

Available methods:

  • addValueInCache(String varName,Object value) - expiration in 1 day

  • addValueInCache(String varName,Object value,Long expirationTime)- expiration expressed in minutes

  • removeValueInCache(String varName)

  • removeValuesFromCache(String keyPrefix) - Remove multiple values from cache: the one whose key starts with the specified pattern.

  • invalidateAll()

  • Object getValueInCache(String varName) - return if the variable is stored in cache, it returns the corresponding value, otherwise an exception will be fired

  • var string = clearCache(String varNamesStr,String keysStr) - Invoke GAE instance and ask for clearing up a list of variable names from the MemCache. Alternatively, if keys is specified as well, it removes only part of it. Arguments:

    • varNames a varName or many variable names separated by a comma (e.g. "MOTPROM,MOTPROM_BANCHES")

    • keys optional: can be null; a key within or many keys, separated by a comma; if specified, the cached entry related to varName must be a map and such a map inside the cache will not be removed completely, but only the entries related to the specified keys

SQL query execution with very long result set

In case of a SQL query returning a very long result set, it is NOT recommended to use the previous method, since it gives back the whole result set at once, which is dangerous because it can consume a large amount of memory on the server.

A better solution is represented by the following method, where the result set is read row by row: for each row, a callback function is invoked, in order to process it.

Consequently, this method cannot be coupled to the user interface, since it would mean that all data would be read in the end. This approach is good when the row processing does not involve the UI, but some other operation on the server side, like writing a text file, starting from the result set.

Syntax

utils.executeQueryWithCallback(
  callbackFunName,
  sql,
  dataStoreId,
  separatedTransaction,
  interruptExecution,
  params
)

Details

Generic SQL execution (NO SQL queries) without logging it

Syntax

var rows = utils.executeSqlNoLog(
sql,
dataSourceId,
separatedTransaction,
interruptExecution,
params
)

Details

Note: the SQL operation will not be logged. This method csan be useful with bulk operations, whose execution could slow down if a log message were produced for each execution.

Generic SQL execution (NO SQL queries)

Syntax

var rows = utils.executeSql(sql, dataSourceId, separatedTransaction, interruptExecution, params)

Details

Execute a GQL query on Google Datastore

The datastore must be already configured as a global parameter. Once done that, it is possible to execute a query statement, in order to fetch a list of entities. The query language is GQL: filtering and sorting conditions are strictly ruled by the Google datastore. That means that additional indexes could be defined before executing the query. For instance, = operators can be used without additional indexes, but it is not so for sorting conditions or filtering conditions having not equal operators (e.g. <, <=, etc.). See Datastore syntax to get detail information about the syntax to use when filtering entities.

Syntax

var json = utils.executeQueryOnGoogleDatastore(gql,dataModelId,interruptExecution, params);

Details

Example

var json = utils.executeQueryOnGoogleDatastore(
"SELECT * FROM Products WHERE companyId=:COMPANY_ID AND enabled='Y' ",
XXX, // XXX must be the data model id
true, // fire an Execution in case of error
[]
);

Note: every GQL instruction will be logged.

Note: in case of a data model where there are attributes having type Array, this method will get back also the array value, expressed as a String whose values are separated by a comma.

Important note: please use the cached version of this method as often as possible:

var json = utils.executeCachedQueryOnGoogleDatastore(maxCachedEntities, gql,dataModelId,interruptExecution, params);

where maxCachedEntities is the max number of cached entities having the same entity name specified in the GQL query.

Insert a single entity into the Google Datastore

The entity is expressed as a Javascript object.

The datastore must be already configured as a global parameter.Once done that, it is possible to execute operations on the Google Datastore.

Syntax

var json = utils.insertObjectOnGoogleDatastore(obj, dataModelId, interruptExecution);

Details

Note: in case of a data model where there are attributes having type Array, this method will get back also the array value, expressed as a String whose values are separated by a comma.

Update a single entity into the Google Datastore

The entity is expressed as a Javascript object

The datastore must be already configured as a global parameter.Once done that, it is possible to execute operations on the Google Datastore.

Syntax

var json = utils.updateObjectOnGoogleDatastore(obj, dataModelId, interruptExecution);

Details

Note: in case of a data model where there are attributes having type Array, this method will get back also the array value, expressed as a String whose values are separated by a comma.

Delete a single entity from the Google Datastore

The entity is expressed as a Javascript object.The datastore must be already configured as a global parameter.Once done that, it is possible to execute operations on the Google Datastore.

Syntax

var json = utils.deleteObjectOnGoogleDatastore(obj, dataModelId, interruptExecution);

Details

ArgumentDescriptionobja Javascript object related to the entity stored in the Datastore and to removedataModelIdit identifies the data model having "datastore" type, related to the entity to removeinterruptExecutionboolean flag used to define if the executing of the current server-side javascript program must be interrupted in case of an error during the execution of the operationoktrue in case of the operation has been executed successfully, an exception otherwise

Get a parameter value

Syntax

var paramValue = utils.getParameter(paramName)

Details

ArgumentDescriptionparamNamea string value representing a parameter named defined at installation level (CON44 table) or at application level (CON07 table)

paramValue the value defined for the specified parameter name or null if not found; if the parameter is found as an application parameter, that value is returned, otherwise it will be returned the value defined at installation level

Get a custom user parameter value

Syntax

var value = utils.getCustomApplUserVars(String varName)

Replace internal representation of a javascript String (CompString) with a String

Syntax

var vo = utils.fixVo(vo)

Execute a javascript GAE action starting from another one

Syntax

var json = utils.executeAction(actionId, vo, params, headers);

Last updated