Google App Engine specific shortcuts

In addition to providing direct access to the App Engine services, Gaelyk also adds some syntax sugar on top of these APIs. Let's review some of these improvements.

Improvements to the low-level datastore API

Although it's possible to use JDO and JPA in Google App Engine, Gaelyk also lets you use the low-level raw API for accessing the datastore, and makes the Entity class from that API a bit more Groovy-friendly.

Using Entitys as maps or POJOs/POGOs

Note: POGO stands for Plain Old Groovy Object.

When you use the Entity class from Java, you have to use methods like setProperty() or getProperty() to access the properties of your Entity, making the code more verbose than it needs to be (at least in Java). Ultimately, you would like to be able to use this class (and its instances) as if they were just like a simple map, or as a normal Java Bean. That's what Gaelyk proposes by letting you use the subscript operator just like on maps, or a normal property notation. The following example shows how you can access Entitys: 

    import com.google.appengine.api.datastore.Entity

    Entity entity = new Entity("person")

    // subscript notation, like when accessing a map
    entity['name'] = "Guillaume Laforge"
    println entity['name']

    // normal property access notation
    entity.age = 32
    println entity.age
Note: For string properties, Google App Engine usually distinguishes between strings longer or shorter than 500 characters. Short strings are just mere Java strings, while longer strings (>500 chars) should be wrapped in a Text instance. Gaelyk shields you from taking care of the difference, and instead, when using the two notations above, you just have to deal with mere Java strings, and don't need to use the Text class at all.

Some properties of your entities can be unindexed, meaning that they can't be used for your search criteria. This may be the case for long text properties, for example a bio of a person, etc. The property notation or subscript notation use normal indexed properties. If you want to set unindexed properties, you can use the unindexed shorcut: 

    import com.google.appengine.api.datastore.Entity

    Entity entity = new Entity("person")

    entity.name = "Guillaume Laforge"
    entity.unindexed.bio = "Groovy Project Manager..."
    entity.unindexed['address'] = "Very long address..."

A handy mechanism exists to assign several properties at once, on your entities, using the << (left shift) operator. This is particularly useful when you have properties coming from the request, in the params map variable. You can the do the following to assign all the key/values in the map as properties on your entity: 

    // the request parameters contain a firstname, lastname and age key/values:
    // params = [firstname: 'Guillaume', lastname: 'Laforge', title: 'Groovy Project Manager']

    Entity entity = new Entity("person")

    entity << params

    assert entity.lastname == 'Laforge'
    assert entity.firstname == 'Guillaume'
    assert entity.title == 'Groovy Project Manager'

    // you can also select only the key/value pairs you'd like to set on the entity
    // thanks to Groovy's subMap() method, which will create a new map with just the keys you want to keep
    entity << params.subMap(['firstname', 'lastname'])
Note: Gaelyk adds a few converter methods to ease the creation of instances of some GAE SDK types that can be used as properties of entities, using the as operator: 
    "foobar@gmail.com" as Email
    "foobar@gmail.com" as JID

    "http://www.google.com" as Link
    new URL("http://gaelyk.appspot.com") as Link

    "+33612345678" as PhoneNumber
    "50 avenue de la Madeleine, Paris" as PostalAddress

    "groovy" as Category

    32 as Rating
    "32" as Rating

    "long text" as Text

    "some byte".getBytes() as Blob
    "some byte".getBytes() as ShortBlob

    "foobar" as BlobKey

    [45.32, 54.54] as GeoPt

Converting beans to entities and back

The mechanism explained above with type conversions (actually called "coercion") is also available and can be handy for converting between a concrete bean and an entity. Any POJO or POGO can thus be converted into an Entity, and you can also convert an Entity to a POJO or POGO. 

    // given a POJO
    class Person {
        String name
        int age
    }

    def e1 = new Entity("Person")
    e1.name = "Guillaume"
    e1.age = 33

    // coerce an entity into a POJO
    def p1 = e1 as Person

    assert e1.name == p1.name
    assert e1.age == p1.age

    def p2 = new Person(name: "Guillaume", age: 33)
    // coerce a POJO into an entity
    def e2 = p2 as Entity

    assert p2.name == e2.name
    assert p2.age == e2.age
Note: The POJO/POGO class simpleName property is used as the entity kind. So for example, if the Person class was in a package com.foo, the entity kind used would be Person, not the fully-qualified name. This is the same default strategy that Objectify is using. You can further customize coercion by implementing DatastoreEntity interface.

Further customization of the coercion can be achieved by using annotations on your classes:

Here's an example of a Person bean using @Entity annotation, whose key is a string login, whose biography should be unindexed, and whose full name can be ignored since it's a computed property: 

    import groovyx.gaelyk.datastore.Entity
    import groovyx.gaelyk.datastore.Key
    import groovyx.gaelyk.datastore.Unindexed
    import groovyx.gaelyk.datastore.Ignore
    
    @Entity(unindexed=false)
    class Person {
        @Key String login
        String firstName
        String lastName
        @Unindexed String bio
        @Ignore String getFullName() { "$firstName $lastName" }
    }

Thanks to @Entity annotation, you get basic CRUD operations for free: 

   
    assert Person.count()   == 0
    
    def glaforge = new Person(
        login:      'glaforge', 
        firstName:  'Guillaume', 
        lastName:   'Laforge',
        bio:        'Groovy Project Manager'
    ).save()
    
    assert Person.count()   == 1
    assert glaforge         == Person.get('glaforge')
    assert Person.findAll { where firstName == 'Guillaume' } == 1
    
    glaforge.delete()
    assert Person.count()   == 0
Note: In turn, with this feature, you have a lightweight object/entity mapper. However, remember it's a simplistic solution for doing object/entity mapping, and this solution doesn't take into accounts relationships and such. If you're really interested in a fully featured mapper, you should have a look at Objectify or Twig.

List to Key conversion

Another coercion mechanism that you can take advantage of, is to use a list to Key conversion, instead of using the more verbose KeyFactory.createKey() methods: 

    [parentKey, 'address', 333] as Key
    [parentKey, 'address', 'name'] as Key
    ['address', 444] as Key
    ['address', 'name'] as Key

Added save() and delete() methods on Entity

In the previous sub-section, we've created an Entity, but we need to store it in Google App Engine's datastore. We may also wish to delete an Entity we would have retrieved from that datastore. For doing so, in a classical way, you'd need to call the save() and put() methods from the DataService instance. However, Gaelyk dynamically adds a save() and delete() method on Entity: 

    def entity = new Entity("person")
    entity.name = "Guillaume Laforge"
    entity.age = 32

    entity.save()

Afterwards, if you need to delete the Entity you're working on, you can simply call: 

    entity.delete()

Added delete() and get() method on Key

Sometimes, you are dealing with keys, rather than dealing with entities directly — the main reaons being often for performance sake, as you don't have to load the full entity. If you want to delete an element in the datastore, when you just have the key, you can do so as follows: 

    someEntityKey.delete()

Given a Key, you can get the associated entity with the get() method: 

    Entity e = someEntityKey.get()

And if you have a list of entities, you can get them all at once: 

    def map = [key1, key2].get()

    // and then access the returned entity from the map:
    map[key1]

Converting Key to an encoded String and vice-versa

When you want to store a Key as a string or pass it as a URL parameter, you can use the KeyFactory methods to encode / decode keys and their string representations. Gaelyk provides two convenient coercion mechanisms to get the encoded string representation of a key: 

    def key = ['addresses', 1234] as Key
    def encodedKey = key as String

And to retrieve the key from its encoded string representation:

    def encodedKey = params.personKey  // the encoded string representation of the key
    def key = encodedKey as Key

Added withTransaction() method on the datastore service

Last but not least, if you want to work with transactions, instead of using the beginTransaction() method of DataService, then the commit() and rollback() methods on that Transaction, and doing the proper transaction handling yourself, you can use the withTransaction() method that Gaelyk adds on DataService and which takes care of that boring task for you: 

    datastore.withTransaction {
        // do stuff with your entities within the transaction
    }
    // enable cross group transactions
    datastore.withTransaction(true) {
        // do stuff with more than one entity group
        new Entity('foo').save()
        new Entity('bar').save()
    }

The withTransaction() method takes a closure as the two parameters. First one is optional boolean value. If set to true cross group transactions are enabled. The second parameter is closure and within that closure, upon its execution by Gaelyk, your code will be in the context of a transaction.

Added get() methods on the datastore service

To retrieve entities from the datastore, you can use the datastore.get(someKey) method, and pass it a Key you'd have created with KeyFactory.createKey(...): this is a bit verbose, and Gaelyk proposes additional get() methods on the datastore service, which do the key creation for you: 

    Key pk = ... // some parent key
    datastore.get(pk, 'address', 'home') // by parent key, kind and name
    datastore.get(pk, 'address', 1234)   // by parent key, kind and id

    datastore.get('animal', 'Felix')     // by kind and name
    datastore.get('animal', 2345)        // by kind and id

This mechanism also works with the asynchronous datastore, as Gaelyk wraps the Future<Entity> transparently, so you don't have to call get() on the future: 

    Key pk = ... // some parent key
    datastore.async.get(pk, 'address', 'home') // by parent key, kind and name
    datastore.async.get(pk, 'address', 1234)   // by parent key, kind and id

    datastore.async.get('animal', 'Felix')     // by kind and name
    datastore.async.get('animal', 2345)        // by kind and id
Note: When you have a Future<Entity> f, when you call f.someProperty, Gaelyk will actually lazily call f.get().someProperty, making the usage of the future transparent. However, note it only works for properties, it doesn't work for method call on futures, where you will have to call get() first. This transparent handling of future properties is working for all Futures, not just Future<Entity>.

Querying

With the datastore API, to query the datastore, the usual approach is to create a Query, prepare a PreparedQuery, and retrieve the results as a list or iterator. Below you will see an example of queries used in the Groovy Web Console to retrieve scripts written by a given author, sorted by descending date of creation: 

    import com.google.appengine.api.datastore.*
    import static com.google.appengine.api.datastore.FetchOptions.Builder.*

    // query the scripts stored in the datastore
    // "savedscript" corresponds to the entity table containing the scripts' text
    def query = new Query("savedscript")

    // sort results by descending order of the creation date
    query.addSort("dateCreated", Query.SortDirection.DESCENDING)

    // filters the entities so as to return only scripts by a certain author
    query.addFilter("author", Query.FilterOperator.EQUAL, params.author)

    PreparedQuery preparedQuery = datastore.prepare(query)

    // return only the first 10 results
    def entities = preparedQuery.asList( withLimit(10) )

Fortunately, Gaelyk provides a query DSL for simplifying the way you can query the datastore. Here's what it looks like with the query DSL: 

    def entities = datastore.execute {
        select all from savedscript
        sort desc by dateCreated
        where author == params.author
        limit 10
    }

Let's have a closer look at the syntax supported by the DSL. There are four methods added dynamically to the datastore: query{},iterate{}, execute{} and build{}. query{} allow you to create a Query that you can use then to prepare a PreparedQuery. iterate{} and execute{} is going further as it executes the query to return a single entity, a list, a count, etc. build{} allow you to create a QueryBuilder you can modify later and call iterate{} and execute{} on it.

Creating queries

You can create a Query with the datastore.query{} method. The closure argument passed to this method supports the verbs select, from, where/and and sort. Here are the various options of those verbs: 

    // select the full entity with all its properties
    select all
    // return just the keys of the entities matched by the query
    select keys
    // return just a few properties (must be indexed)
    select name: String, age: Integer
    // return just a few properties (must be indexed) as RawValue
    select name, age

    // specify the entity kind to search into
    from entityKind

    // specify that entities searched should be child of another entity
    // represented by its key
    ancestor entityKey

    // add a filter operation
    // operators allowed are: <, <=, ==, !=, >, >=, in
    where propertyName <  value
    where propertyName <= value
    where propertyName == value
    where propertyName != value
    where propertyName >= value
    where propertyName >  value
    where propertyName in listOfValues

    // you can use "and" instead of "where" to add more where clauses

    // ascending sorting
    sort asc  by propertyName
    // descending sorting
    sort desc by propertyName

    // automatic pagination using limit, offset or cursor parameters
    paginate params
Notes:

Executing queries

You can use the datastore.execute{} call to execute the queries, or the datastore.iterate{} call if you want to get the results in the form of an iterator. The select verb also provides additional values. The from verb allows to specify a class to coerce the results to a POGO. In addition, you can specify the FetchOptions with additional verbs like: limit, offset, range, chunkSize, fetchSize startAt, endAt. If your are facing queries that expires, use restart automatically. This option only works with iterator methdod to for flawless iteration. 

    // select the full entity with all its properties
    select all
    // return just the keys of the entities matched by the query
    select keys
    // return one single entity if the query really returns one single result
    select single
    // return the count of entities matched by the query
    select count
    // return just a few properties (must be indexed)
    select name: String, age: Integer
    // return just a few properties (must be indexed) as RawValue
    select name, age

    // from an entity kind
    from entityKind
    // specify the entity kind as well as a type to coerce the results to
    from entityKind as SomeClass

    // specify that entities searched should be child of another entity
    // represented by its key
    ancestor entityKey

    where propertyName <  value
    where propertyName <= value
    where propertyName == value
    where propertyName != value
    where propertyName >= value
    where propertyName >  value
    where propertyName in listOfValues

    // you can use "and" instead of "where" to add more where clauses

    // ascending sorting
    sort asc  by propertyName
    // descending sorting
    sort desc by propertyName

    // limit to only 10 results
    limit 10
    // return the results starting from a certain offset
    offset 100
    // range combines offset and limit together
    range 100..109

    // fetch and chunk sizes
    fetchSize 100
    chunkSize 100

    // cursor handling
    startAt cursorVariable
    startAt cursorWebSafeStringRepresentation
    endAt cursorVariable
    endAt cursorWebSafeStringRepresentation
    
    // automatically restart query when expired
    // usefull for long running queries
    restart automatically

    // automatic pagination using limit, offset or cursor parameters
    paginate params
Notes: If you use the from addresses as Address clause, specifying a class to coerce the results into, if your where and and clauses use properties that are not present in the target class, a QuerySyntaxException will be thrown.

For select all or select keys queries using iterate or execute the methods return instance of QueryResultIterator or QueryResultList from which cursor and indexList properites could be read.

Asynchronous datastore

In addition to the "synchronous" datastore service, the App Engine SDK also provides an AsynchrnousDatastoreService. You can retrieve the asynchronous service with the datastore.async shortcut.

Gaelyk adds a few methods on entities and keys that leverage the asynchronous service:

Datastore metadata querying

The datastore contains some special entities representing useful metadata, like the available kinds, namespaces and properties. Gaelyk provides shortcuts to interrogate the datastore for such entity metadata.

Namespace querying

    // retrieve the list of namespaces (as a List<Entity>)
    def namespaces = datastore.namespaces

    // access the string names of the namespaces
    def namespaceNames = namespaces.key.name

    // if you want only the first two
    datastore.getNamespaces(FetchOptions.Builder.withLimit(2))

    // if you want to apply further filtering on the underlying datastore query
    datastore.getNamespaces(FetchOptions.Builder.withLimit(2)) { Query query ->
        // apply further filtering on the query parameter
    }

Kind querying

    // retrieve the list of entity kinds (as a List<Entity>)
    def kinds = datastore.kinds

    // get only the string names
    def kindNames = kinds.key.name

    // get the first kind
    datastore.getKinds(FetchOptions.Builder.withLimit(10))

    // futher query filtering:
    datastore.getKinds(FetchOptions.Builder.withLimit(10)) { Query query ->
        // apply further filtering on the query parameter
    }

Properties querying

    // retrieve the list of entity properties (as a List<Entity>)
    def props = datastore.properties

    // as for namespaces and kinds, you can add further filtering
    datastore.getProperties(FetchOptions.Builder.withLimit(10)) { Query query ->
        // apply further filtering on the query parameter
    }

    // if you want to retrive the list of properties for a given entity kind,
    // for an entity Person, with two properties name and age:
    def entityKindProps = datastore.getProperties('Person')
    // lists of entity names
    assert entityKindProps.key.parent.name == ['Person', 'Person']
    // list of entity properties
    assert entityKindProps.key.name == ['name', 'age']

The task queue API shortcuts

Google App Engine SDK provides support for "task queues". An application has a default queue, but other queues can be added through the configuration of a queue.xml file in /WEB-INF.

Note: You can learn more about queues and task queues, and how to configure them on the online documentation.

In your Groovlets and templates, you can access the default queue directly, as it is passed into the binding: 

    // access the default queue
    defaultQueue

You can access the queues either using a subscript notation or the property access notation: 

    // access a configured queue named "dailyEmailQueue" using the subscript notation
    queues['dailyEmailQueue']

    // or using the property access notation
    queues.dailyEmailQueue

    // you can also access the default queue with:
    queues.default

To get the name of a queue, you can call the provided getQueueName() method, but Gaelyk provides also a getName() method on Queue so that you can write queue.name, instead of the more verbose queue.getQueueName() or queue.queueName, thus avoid repetition of queue.

For creating tasks and submitting them on a queue, with the SDK you have to use the TaskOptions.Builder. In addition to this builder approach, Gaelyk provides a shortcut notation for adding tasks to the queue using named arguments: 

    // add a task to the queue synchronously
    TaskHandle handle = queue.add countdownMillis: 1000, url: "/task/dailyEmail",
        taskName: "dailyNewsletter",
        method: 'PUT', params: [date: '20101214'],
        payload: content, retryOptions: RetryOptions.Builder.withDefaults()
        
    // add a task to the queue asynchronously
    Future<TaskHandle> future = queue.addAsync countdownMillis: 1000, url: "/task/dailyEmail",
        taskName: "dailyNewsletter",
        method: 'PUT', params: [date: '20101214'],
        payload: content, retryOptions: RetryOptions.Builder.withDefaults()

There is also a variant with an overloaded << operator for the second one: 

    // add a task to the queue
    queue << [
        countdownMillis: 1000, url: "/task/dailyEmail",
        taskName: "dailyNewsletter",
        method: 'PUT', params: [date: '20101214'],
        payload: content,
        retryOptions: [
            taskRetryLimit: 10,
            taskAgeLimitSeconds: 100,
            minBackoffSeconds: 40,
            maxBackoffSeconds: 50,
            maxDoublings: 15
        ]
    ]

Email support

New send() method for the mail service

Gaelyk adds a new send() method to the mail service, which takes named arguments. That way, you don't have to manually build a new message yourself. In your Groovlet, for sending a message, you can do this: 

    mail.send from: "app-admin-email@gmail.com",
            to: "recipient@somecompany.com",
            subject: "Hello",
            textBody: "Hello, how are you doing? -- MrG",
            attachment: [data: "Chapter 1, Chapter 2".bytes, fileName: "outline.txt"]

Similarily, a sendToAdmins() method was added to, for sending emails to the administrators of the application.

Note: There is a sender alias for the from attribute. And instead of a textBody attribute, you can send HTML content with the htmlBody attribute.
Note: There are two attachment attributes: attachment and attachments.

Incoming email messages

Your applications can also receive incoming email messages, in a similar vein as the incoming XMPP messaging support. To enable incoming email support, you first need to update your appengine-web.xml file as follows: 

    <inbound-services>
        <service>mail</service>
    </inbound-services>

In your web.xml file, you can eventually add a security constraint on the web handler that will take care of treating the incoming emails: 

    ...
    <!-- Only allow the SDK and administrators to have access to the incoming email endpoint -->
    <security-constraint>
        <web-resource-collection>
            <url-pattern>/_ah/mail/*</url-pattern>
        </web-resource-collection>
        <auth-constraint>
            <role-name>admin</role-name>
        </auth-constraint>
    </security-constraint>
    ...

You need to define a Groovlet handler for receiving the incoming emails with a special route definition, in your /WEB-INF/routes.groovy configuration file: 

    email to: "/receiveEmail.groovy"
Remark: You are obviously free to change the name and path of the Groovlet.

All the incoming emails will be sent as MIME messages through the request of your Groovlet. To parse the MIME message, you'll be able to use the parseMessage(request) method on the mail service injected in the binding of your Groovlet, which returns a javax.mail.MimeMessage instance: 

    def msg = mail.parseMessage(request)

    log.info "Subject ${msg.subject}, to ${msg.allRecipients.join(', ')}, from ${msg.from[0]}"

XMPP/Jabber support

Your application can send and receive instant messaging through XMPP/Jabber.

Note: You can learn more about XMPP support on the online documentation.

Sending messages

Gaelyk provides a few additional methods to take care of sending instant messages, get the presence of users, or to send invitations to other users. Applications usually have a corresponding Jabber ID named after your application ID, such as yourappid@appspot.com. To be able to send messages to other users, your application will have to invite other users, or be invited to chat. So make sure you do so for being able to send messages.

Let's see what it would look like in a Groovlet for sending messages to a user: 

    String recipient = "someone@gmail.com"

    // check if the user is online
    if (xmpp.getPresence(recipient).isAvailable()) {
        // send the message
        def status = xmpp.send(to: recipient, body: "Hello, how are you?")

        // checks the message was successfully delivered to all the recipients
        assert status.isSuccessful()
    }

Gaelyk once again decorates the various XMPP-related classes in the App Engine SDK with new methods:

To give you a little more details on the various attributes you can use to create messages to be sent, you can pass the following attributes to the send() method of XMPPService:

Note: body and xml are exclusive, you can't specify both at the same time.

We mentioned the ability to send XML payloads, instead of normal chat messages: this functionality is particularly interesting if you want to use XMPP/Jabber as a communication transport between services, computers, etc. (ie. not just real human beings in front of their computer). We've shown an example of sending raw text messages, here's how you could use closures in the xml to send XML fragments to a remote service: 

    String recipient = "service@gmail.com"

    // check if the service is online
    if (xmpp.getPresence(recipient).isAvailable()) {
        // send the message
        def status = xmpp.send(to: recipient, xml: {
            customers {
                customer(id: 1) {
                    name 'Google'
                }
            }
        })

        // checks the message was successfully delivered to the service
        assert status.isSuccessful()
    }
Implementation detail: the closure associated with the xml attribute is actually passed to an instance of StreamingMarkupBuilder which creates an XML stanza.

Receiving messages

It is also possible to receive messages from users. For that purpose, Gaelyk lets you define a Groovlet handler that will be receiving the incoming messages. To enable the reception of messages, you'll have to do two things:

As a first step, let's configure appengine-web.xml by adding this new element:

    <inbound-services>
        <service>xmpp_message</service>
    </inbound-services>

Similarily to the incoming email support, you can define security constraints: 

    ...
    <!-- Only allow the SDK and administrators to have access to the incoming jabber endpoint -->
    <security-constraint>
        <web-resource-collection>
            <url-pattern>/_ah/xmpp/message/chat/</url-pattern>
        </web-resource-collection>
        <auth-constraint>
            <role-name>admin</role-name>
        </auth-constraint>
    </security-constraint>
    ...

Then let's add the route definition in routes.groovy:

    jabber to: "/receiveJabber.groovy"

Alternatively, you can use the longer version: 

    jabber chat, to: "/receiveJabber.groovy"
Remark: You are obviously free to change the name and path of the Groovlet.

All the incoming Jabber/XMPP messages will be sent through the request of your Groovlet. Thanks to the parseMessage(request) method on the xmpp service injected in the binding of your Groovlet, you'll be able to access the details of a Message instance, as shown below: 

    def message = xmpp.parseMessage(request)

    log.info "Received from ${message.from} with body ${message.body}"

    // if the message is an XML document instead of a raw string message
    if (message.isXml()) {
        // get the raw XML string
        message.stanza

        // or get a document parsed with XmlSlurper
        message.xml()
    }

XMPP presence handling

To be notified of users' presence, you should first configure appengine-web.xml to specify you want to activate the incoming presence service: 

    <inbound-services>
        <service>xmpp_presence</service>
    </inbound-services>

Then, add a special route definition in routes.groovy: 

    jabber presence, to: "/presence.groovy"
Remark: You are obviously free to change the name and path of the Groovlet handling the presence requests.

Now, in your presence.groovy Groovlet, you can call the overriden XMPPService#parsePresence method: 

    // parse the incoming presence from the request
    def presence = xmpp.parsePresence(request)

    log.info "${presence.fromJid.id} is ${presence.available ? '' : 'not'} available"

XMPP subscription handling

To be notified of subscriptions, you should first configure appengine-web.xml to specify you want to activate the incoming subscription service: 

    <inbound-services>
        <service>xmpp_subscribe</service>
    </inbound-services>

Then, add a special route definition in routes.groovy: 

    jabber subscription, to: "/subscription.groovy"
Remark: You are obviously free to change the name and path of the Groovlet handling the subscription requests.

Now, in your subscription.groovy Groovlet, you can call the overriden XMPPService#parseSubscription method: 

    // parse the incoming subscription from the request
    def subscription = xmpp.parseSubscription(request)

    log.info "Subscription from ${subscription.fromJid.id}: ${subscription.subscriptionType}}"

Enhancements to the Memcache service

Gaelyk provides a few additional methods to the Memcache service, to get and put values in the cache using Groovy's natural subscript notation, as well as for using the in keyword to check when a key is present in the cache or not. 

    // under src/main/groovy
    class Country implements Serializable {
        static final long serialVersionUID = 123456L;
        String name 
    }

	// in groovlet
    def countryFr = new Country(name: 'France')

    // use the subscript notation to put a country object in the cache, identified by a string
    // (you can also use non-string keys)
    memcache['FR'] = countryFr

    // check that a key is present in the cache
    if ('FR' in memcache) {
        // use the subscript notation to get an entry from the cache using a key
        def countryFromCache = memcache['FR']
    }
Note: Make sure the objects you put in the cache are serializable. Also, be careful with the last example above as the 'FR' entry in the cache may have disappeared between the time you do the if (... in ...) check and the time you actually retrieve the value associated with the key from memcache.

Asynchronous Memcache service

The Memcache service is synchronous, but App Engine also proposes an asynchronous Memcache service that you can access by calling the async property on the Memcache service instance: 

    memcache.async.put(key, value)
Note: Additionally, the usual property notation and subscript access notation are also available.

Closure memoization

As Wikipedia puts it, memoization is an optimization technique used primarily to speed up computer programs by having function calls avoid repeating the calculation of results for previously-processed inputs. Gaelyk provides such a mechanism for closures, storing invocation information (a closure call with its arguments values) in memcache.

An example, if you want to avoid computing expansive operations (like repeatedly fetching results from the datastore) in a complex algorithm: 

    Closure countEntities = memcache.memoize { String kind ->
        datastore.prepare( new Query(kind) ).countEntities()
    }

    // the first time, the expensive datastore operation will be performed and cached
    def totalPics = countEntities('photo')

    /* add new pictures to the datastore */

    // the second invocation, the result of the call will be the same as before, coming from the cache
    def totalPics2 = countEntities('photo')
Note: Invocations are stored in memcache only for up to the 60 seconds request time limit of App Engine.

Enhancements related to the Blobstore and File services

Gaelyk provides several enhancements around the usage of the blobstore service.

Getting blob information

Given a blob key, you can retrieve various details about the blob when it was uploaded: 

    BlobKey blob = ...

    // retrieve an instance of BlobInfo
    BlobInfo info = blob.info

    // directly access the BlobInfo details from the key itself
    String filename     = blob.filename
    String contentType  = blob.contentType
    Date creation       = blob.creation
    long size           = blob.size

Serving blobs

With the blobstore service, you can stream the content of blobs back to the browser, directly on the response object: 

    BlobKey blob = ...

    // serve the whole blob
    blob.serve response

    // serve a fragment of the blob
    def range = new ByteRange(1000) // starting from 1000
    blob.serve response, range

    // serve a fragment of the blob using an int range
    blob.serve response, 1000..2000

Reading the content of a Blob

Beyond the ability to serve blobs directly to the response output stream with blobstoreService.serve(blobKey, response) from your groovlet, there is the possibility of obtaining an InputStream to read the content of the blob. Gaelyk adds three convenient methods on BlobKey to easily deal with a raw input stream or with a reader, leveraging Groovy's own input stream and reader methods. The stream and reader are handled properly with regards to cleanly opening and closing those resources so that you don't have to take care of that aspect yourself. 

    BlobKey blobKey = ...

    blobKey.withStream { InputStream stream ->
        // do something with the stream
    }

    // defaults to using UTF-8 as encoding for reading from the underlying stream
    blobKey.withReader { Reader reader ->
        // do something with the reader
    }

    // specifying the encoding of your choice
    blobKey.withReader("UTF-8") { Reader reader ->
        // do something with the reader
    }

You can also fetch byte arrays for a given range: 

    BlobKey blob = ...
    byte[] bytes

    // using longs
    bytes = blob.fetchData 1000, 2000

    // using a Groovy int range
    bytes = blob.fetchData 1000..2000

    // using a ByteRange
    def range = new ByteRange(1000, 2000) // or 1000..2000 as ByteRange
    bytes = blob.fetchData range

Deleting a blob

Given a blob key, you can easily delete it thanks to the delete() method:

    BlobKey blob = ...

    blob.delete()

Iterating over and collecting BlobInfos

The blobstore service stores blobs that are identified by BlobKeys, and whose metadata are represented by BlobInfo. If you want to iterate over all the blobs from the blobstore, you can use the BlobInfoFactory and its queryBlobInfos() method, but Gaelyk simplifies that job with an each{} and a collect{} method right from the blobstore service: 

    blobstore.each { BlobInfo info -> out << info.filename }

    def fileNames = blobstore.collect { BlobInfo info -> info.filename }

Example Blobstore service usage

In this section, we'll show you a full-blown example. First of all, let's create a form to submit a file to the blobstore, in a template named upload.gtpl at the root of your war: 

    <html>
    <body>
        <h1>Please upload a text file</h1>
        <form action="${blobstore.createUploadUrl('/uploadBlob.groovy')}"
                method="post" enctype="multipart/form-data">
            <input type="file" name="myTextFile">
            <input type="submit" value="Submit">
        </form>
    </body>
    </html>

The form will be posted to a URL created by the blobstore service, that will then forward back to the URL you've provided when calling blobstore.createUploadUrl('/uploadBlob.groovy')

Warning: The URL to he groovlet to which the blobstore service will forward the uploaded blob details should be a direct path to the groovlet like /uploadBlob.groovy. For an unknown reason, you cannot use a URL defined through the URL routing system. This is not necessarily critical, in the sense that this URL is never deployed in the browser anyway.

Now, create a groovlet named uploadBlob.groovy stored in /WEB-INF/groovy with the following content: 

    def blobs = blobstore.getUploadedBlobs(request)
    def blob = blobs["myTextFile"]

    response.status = 302

    if (blob) {
        redirect "/success?key=${blob.keyString}"
    } else {
        redirect "/failure"
    }

In the groovlet, you retrieve all the blobs uploaded in the upload.gtpl page, and more particularly, the blob coming from the myTextFile input file element.

Warning: Google App Engine mandates that you explicitly specify a redirection status code (301, 302 or 303), and that you do redirect the user somewhere else, otherwise you'll get some runtime errors.

We define some friendly URLs in the URL routing definitions for the upload form template, the success and failure pages: 

    get "/upload",  forward: "/upload.gtpl"
    get "/success", forward: "/success.gtpl"
    get "/failure", forward: "/failure.gtpl"

You then create a failure.gtpl page at the root of your war directory: 

    <html>
        <body>
            <h1>Failure</h1>
            <h2>Impossible to store or access the uploaded blob</h2>
        </body>
    </html>

And a success.gtpl page at the root of your war directory, showing the blob details, and outputing the content of the blob (a text file in our case): 

    <% import com.google.appengine.api.blobstore.BlobKey %>
    <html>
        <body>
            <h1>Success</h1>
            <% def blob = new BlobKey(params.key) %>

            <div>
                File name: ${blob.filename} <br/>
                Content type: ${blob.contentType}<br/>
                Creation date: ${blob.creation}<br/>
                Size: ${blob.size}
            </div>

            <h2>Content of the blob</h2>

            <div>
                <% blob.withReader { out << it.text } %>
            </div>
        </body>
    </html>

Now that you're all set up, you can access http://localhost:8080/upload, submit a text file to upload, and click on the button. Google App Engine will store the blob and forward the blob information to your uploadBlob.groovy groovlet that will then redirect to the success page (or failure page in case something goes wrong).

File service

The File service API provides a convenient solution for accessing the blobstore, and particularly for programmatically adding blobs without having to go through the blobstore form-based upload facilities. Gaelyk adds a files variable in the binding of Groovlets and templates, which corresponds to the FileService instance.

Writing text content

Inspired by Groovy's own withWriter{} method, a new method is available on AppEngineFile that can be used as follows, to write text content through a writer: 

    // let's first create a new blob file through the regular FileService method
    def file = files.createNewBlobFile("text/plain", "hello.txt")

    file.withWriter { writer ->
        writer << "some content"
    }

You can also specify three options to the withWriter{} method, in the form of named arguments: 

    file.withWriter(encoding: "US-ASCII", locked: false, finalize: false) { writer ->
        writer << "some content"
    }

Writing binary content

In a similar fashion, you can write to an output stream your binary content:

    // let's first create a new blob file through the regular FileService method
    def file = files.createNewBlobFile("text/plain", "hello.txt")

    file.withOutputStream { stream ->
        stream << "Hello World".bytes
    }

You can also specify two options to the withOutputStream{} method, in the form of named arguments: 

    file.withOutputStream(locked: false, finalize: false) { writer ->
        writer << "Hello World".bytes
    }
Note: To finalize a file in the blobstore, App Engine mandates the file needs to be locked. That's why by default locked and finalize are set to true by default. When you want to later be able to append again to the file, make sure to set finalize to false. And if you want to avoid others from concurrently writing to your file, it's better to set locked to true.

Reading binary content

Gaelyk already provides reading capabilities from the blobstore support, as we've already seen, but the File service also supports reading from AppEngineFiles. To read from an AppEngineFile instance, you can use the withInputStream{} method, which takes an optional map of options, and a closure whose argument is a BufferedInputStream: 

    file.withInputStream { BufferedInputStream stream ->
        // read from the stream
    }

You can also specify an option for locking the file (the file is locked by default): 

    file.withInputStream(locked: false) { BufferedInputStream stream ->
        // read from the stream
    }

Reading text content

Similarily to reading from an input stream, you can also read from a BufferedReader, with the withReader{} method: 

    file.withReader { BufferedReader reader ->
        log.info reader.text
    }

You can also specify an option for locking the file (the file is locked by default): 

    file.withReader(locked: false) { BufferedReader reader ->
        log.info reader.text
    }

Miscelanous improvements

If you store a file path in the form of a string (for instance for storing its reference in the datastore), you need to get back an AppEngineFile from its string representation: 

    def path = someEntity.filePath
    def file = files.fromPath(path)

If you have a BlobKey, you can retrieve the associated AppEngineFile: 

    def key = ... // some BlobKey
    def file = key.file

You can retrieve the blob key associated with your file (for example when you want to access an Image instance: 

    def key = file.blobKey
    def image = key.image

And if you want to delete a file without going through the blobstore service, you can do: 

    file.delete()

Namespace support

Google App Engine SDK allows you to create "multitenant"-aware applications, through the concept of namespace, that you can handle through the NamespaceManager class.

Gaelyk adds the variable namespace into the binding of your groovlets and templates. This namespace variable is simply the NamespaceManager class. Gaelyk adds a handy method for automating the pattern of setting a temporary namespace and restoring it to its previous value, thanks to the added of() method, taking a namespace name in the form of a string, and a closure to be executed when that namespace is active. This method can be used as follows: 

    // temporarily set a new namespace
    namespace.of("customerA") {
        // use whatever service leveraging the namespace support
        // like the datastore or memcache
    }
    // once the closure is executed, the old namespace is restored

Images service enhancements

The images service and service factory wrapper

The Google App Engine SDK is providing two classes for handling images:

Very quickly, as you use the images handling capabilities of the API, you quickly end up jumping between the factory and the service class all the time. But thanks to Gaelyk, both ImagesServiceFactory and ImagesService are combined into one. So you can call any method on either of them on the same images instance available in your groovlets and templates. 

    // retrieve an image stored in the blobstore
    def image = images.makeImageFromBlob(blob)

    // apply a resize transform on the image to create a thumbnail
    def thumbnail = images.applyTransform(images.makeResize(260, 260), image)

    // serve the binary data of the image to the servlet output stream
    sout << thumbnail.imageData

On the first line above, we created the image out of the blobstore using the images service, but there is also a more rapid shortcut for retrieving an image when given a blob key: 

    def blobKey = ...
    def image = blobKey.image
Note: blobKey.image creates an image object with only the blob key set. It's not retrieving the actual image right away, nor its properties like its dimensions. See this Gaelyk issue for more information, or this Google App Engine issue.

In case you have a file or a byte array representing your image, you can also easily instanciate an Image with: 

    // from a byte array
    byte[] byteArray = ...
    def image = byteArray.image

    // from a file directly
    image = new File('/images/myimg.png').image

An image manipulation language

The images service permits the manipulation of images by applying various transforms, like resize, crop, flip (vertically or horizontally), rotate, and even an "I'm feeling lucky" transform! The Gaelyk image manipulation DSL allows to simplify the combination of such operations= 

    blobKey.image.transform {
        resize 100, 100
        crop 0.1, 0.1, 0.9, 0.9
        horizontal flip
        vertical flip
        rotate 90
        feeling lucky
    }

The benefit of this approach is that transforms are combined within a single composite transform, which will be applied in one row to the original image, thus saving on CPU computation. But if you just need to make one transform, you can also call new methods on Image as follows: 

    def image = ...

    def thumbnail   = image.resize(100, 100)
    def cropped     = image.crop(0.1, 0.1, 0.9, 0.9)
    def hmirror     = image.horizontalFlip()
    def vmirror     = image.verticalFlip()
    def rotated     = image.rotate(90)
    def lucky       = image.imFeelingLucky()

Capabilities service support

Occasionally, Google App Engine will experience some reliability issues with its various services, or certain services will be down for scheduled maintenance. The Google App Engine SDK provides a service, the CapabilitiesService, to query the current status of the services. Gaelyk adds support for this service, by injecting it in the binding of your groovlets and templates, and by adding some syntax sugar to simplify its use. 

    import static com.google.appengine.api.capabilities.Capability.*
    import static com.google.appengine.api.capabilities.CapabilityStatus.*

    if (capabilities[DATASTORE] == ENABLED && capabilities[DATASTORE_WRITE] == ENABLED) {
        // write something into the datastore
    } else {
        // redirect the user to a page with a nice maintenance message
    }
Note: Make sure to have a look at the capability-aware URL routing configuration.

The services that can be queried are defined as static constants on Capability and currently are:

The different possible statuses are defined in the CapabilityStatus enum:

Tip: Make sure to static import Capability and CapabilityStatus in order to keep your code as concise and readable as possible, like in the previous example, with: 
    import static com.google.appengine.api.capabilities.Capability.*
    import static com.google.appengine.api.capabilities.CapabilityStatus.*

Additionally, instead of comparing explicitely against a specific CapabilityStatus, Gaelyk provides a coercion of the status to a boolean (also called "Groovy Truth"). This allows you to write simpler conditionals: 

    import static com.google.appengine.api.capabilities.Capability.*
    import static com.google.appengine.api.capabilities.CapabilityStatus.*

    if (capabilities[DATASTORE] && capabilities[DATASTORE_WRITE]) {
        // write something into the datastore
    } else {
        // redirect the user to a page with a nice maintenance message
    }
Note: Only the ENABLED and SCHEDULED_MAINTENACE statuses are considered to be true, whereas all the other statuses are considered to be false.

URLFetch Service improvements

Google App Engine offers the URLFetch Service to interact with remote servers, to post to or to fetch content out of external websites. Often, using the URL directly with Groovy's getBytes() or getText() methods is enough, and transparently uses the URLFetch Service under the hood. But sometimes, you need a bit more control of the requests you're making to remote servers, for example for setting specific headers, for posting custom payloads, making asynchronous requests, etc. Gaelyk 0.5 provides a convenient integration of the service with a groovier flavor.

Note: You may also want to have a look at HTTPBuilder's HttpURLClient for a richer HTTP client library that is compatible with Google App Engine.

Gaelyk decorates the URL class with 5 new methods, for the 5 HTTP methods GET, POST, PUT, DELETE, HEAD which can take an optional map for customizing the call:

Those methods return an HTTPResponse or a Future<HTTPResponse> if the async option is set to true.

Let's start with a simple example, say, you want to get the Gaelyk home page content: 

    URL url = new URL('http://gaelyk.appspot.com')

    def response = url.get()

    assert response.responseCode == 200
    assert response.text.contains('Gaelyk')

As you can see above, Gaelyk adds a getText() and getText(String encoding) method to HTTPResponse, so that it is easier to get textual content from remote servers — HTTPResponse only provided a getContent() method that returns a byte array.

If you wanted to make an asynchronous call, you could do: 

    def future = url.get(async: true)
    def response = future.get()

Allowed options

Several options are allowed as arguments of the 5 methods.

To finish on the URLFetch Service support, we can have a look at another example using some of the options above: 

    URL googleSearch = "http://www.google.com/search".toURL()
    HTTPResponse response = googleSearch.get(params: [q: 'Gaelyk'],
            headers: ['User-Agent': 'Mozilla/5.0 (Linux; X11)'])

    assert response.statusCode == 200
    assert response.text.contains('http://gaelyk.appspot.com')
    assert response.headersMap.'Content-Type' == 'text/html; charset=utf-8'
Note: response.statusCode is a synonym of response.responseCode. And notice the convenient response.headersMap shortcut which returns a convenient Map<String, String> of headers instead of SDK's response.headers's List<HTTPHeader>.

Channel Service improvements

For your Comet-style applications, Google App Engine provides its Channel service. The API being very small, beyond adding the channel binding variable, Gaelyk only adds an additional shortcut method for sending message with the same send name as Jabber and Email support (for consistency), but without the need of creating an instance of ChannelMessage: 

    def clientId = "1234"
    channel.createChannel(clientId)
    channel.send clientId, "hello"

Search service support

The full-text search functionality can be accessed with the search variable in the binding of Groovlets and templates. You can also specify a special namespace to restrict the searches to that namespace. 

    // access the search service
    search

    // access the search service for a specific namespace
    search['myNamespace']

You access a particular search index with the index() method, where you specify the index name and consistency mode by passing their values as parameters: 

    def index = search.index("books")

To add documents to an index, you call the put() method on the index, which takes a closure that accepts document(map) {} method calls. You can specify several documents in a single put() call, by simply making several document() calls inside the closure passed to put(). 

    def index = search.index("books")

    def response = index.put {
        document(id: "1234", locale: US, rank: 3) {
            title text: "Big bad wolf", locale: ENGLISH
            published date: new Date()
            numberOfCopies number: 35
            summary html: "
super story
", locale: ENGLISH
            description text: "a book for children"
            category atom: "children"
            category atom: "book"
            keyword text: ["wolf", "red hook"]
            location geoPoint: [15,50]
        }
        // other documents with other document(...) {} calls
    }

The named parameters passed to the document() methods can be id, locale and rank. Inside the closure, you can have as many field definitions of the form: fieldName type: value or with an optional locale: fieldName type: value, locale: someLocale. Fields can be repeated in order to have multi-valued document fields or you can specify map values as list: fieldName type: [one, two]. Empty lists and null values are ignored completely. Such fields are not added to the document.

Once you have added documents to an index, you can search for them, and iterate over all the results: 

    // search the index
    def results = index.search("wolf")

    // iterate over all the resuts
    results.each { ScoredDocument doc ->
        assert doc.id == "1234"

        assert doc.title == "Big bad wolf"
        assert doc.numberOfCopies == 35
        assert doc.summary.contains("story")

        assert doc.keyword.size() == 2
        assert "wolf" in doc.keyword
        assert "red hook" in doc.keyword
    }

As you can see, you can access a document field with the Groovy property notation. When a field is multivalued, the doc.field property access actually returns a list of values.

Because search API is sometimes too much faulty, you can specify number of retries used by index.searchAsync. Following code will attempt to search books related to wolfs three times before failing 

    def results = index.searchAsync("wolf", 3).get()
Note: You can make any closure returning Future retrying by using numberOfRetries * { future } notation. For example 3 * { index.searchAsync("wolf") } will behave the same way as described above. Keep the code inside the closure reasonable small because the closure is called at the beginning of each attempt.

Advanced Full Text Search

App Engine Search API is very unfriendly. It overuses builders in any possible way. Fortunately, Gaelyk provides a search DSL for simplifying the way you running full text queries. The DSL is very close to the datastore query DSL: 

    def documents = search.search {
        select all from books
        sort desc by published, SearchApiLimits.MINIMUM_DATE_VALUE
        where title =~ params.title
        and keyword = params.keyword
        limit 10
    }

The query DSL could be used with two methods on search service object: search and searchAsync. Let's have a closer look at the syntax supported by the DSL: 

    // select the full document with all its fields
    select all
    // return just the ids of the documents matched by the query
    select ids
    // return just a few document's fields
    select name, age
    // return just a few expresions, see https://developers.google.com/appengine/docs/java/search/overview#Expressions
    // methods like distance or geopoint are also supported
    select numberOfCopies: numberOfCopies - 10, body: snippet(params.body, body), rating: max(rating, 10)


    // specify the index to search into
    from books

    // add a filter operation
    // operators allowed are: <, <=, ==, !=, >, >=, =~, ~
    // date values are properly handled for you
    where propertyName <  expression
    where propertyName <= expression
    where propertyName == expression
    where propertyName != expression
    where propertyName >= expression
    where propertyName >  expression
    
    // instead of query operator ':' use Groovy matches operator "=~""
    where propertyName =~ value
    
    // to search for singular and plural form of the word you can use "~" operator 
    where propertyName == ~value

    // you can use "and" instead of "where" to add more where clauses
    // to use logical disjunction ("or"), you can use "||" logical operator
    // you can use "&&" as well for "and"
    where propertyName == value || propertyName == other
    
    // you can also use built-in methods such as distance, geopoint, max, min, count
    where distance(geopoint(10, 50), locality) < 10

    // ascending sorting, the default value is mandatory
    sort asc  by propertyName, defaultValue
    // descending sorting, the default value is mandatory
    sort desc by propertyName, defaultValue
     
    // limit to only 10 results
    limit 10
    // return the results starting from a certain offset
    offset 100

    // cursor handling
    startAt cursorVariable
    startAt cursorWebSafeStringRepresentation
    
    // limits sorting
    limit sort to 1000
    
    // sets number found accuracy to 150
    number found accuracy 150
Notes: