Deploy Go app to App Engine
I previously created a go app accessing a google calendar.
In this post I describe how to deploy it in App Engine. I follow Google Cloud documentation.
Deploy a Go App
I already had installed Cloud SDK in order to deploy my Django app. I updated the installation using sudo apt-get update and sudo apt-get upgrade.
I created my project "indivision-toulon" by running gcloud projects create indivision-toulon --set-as-default. I then checked that everything went well with gcloud projects describe indivision-toulon. Finally I initialized it with a region (I am not sure which one I should choose, and how much it matters in term of pricing between London, Belgium and Zurich): gcloud app create --project=indivision-toulon.
Before going on to follow the doc to build a go app on App Engine, I install the components that will allow to deploy the Go app: sudo apt-get install google-cloud-sdk-app-engine-go.
I added an app.yaml next to my existing go app, containing the following configuration:
runtime: go115
I modified my app so that it does not access Google Calendar for the time being.
Hello World app (Click to expand)
Hello World app (Click to expand)
package main
import (
"fmt"
"log"
"net/http"
"os"
)
// indexHandler responds to requests with our greeting.
func indexHandler(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/" {
http.NotFound(w, r)
return
}
fmt.Fprint(w, "Hello, World!")
}
func main() {
http.HandleFunc("/", indexHandler)
port := os.Getenv("PORT")
if port == "" {
port = "8080"
log.Printf("Defaulting to port %s", port)
}
log.Printf("Listening on port %s", port)
if err := http.ListenAndServe(":"+port, nil); err != nil {
log.Fatal(err)
}
}
Important: it is better to add a .gcloudignore file: similar to a .gitignore file, it will ignore all the files not needed to deploy the app.
It is created in the first steps of the deployment, which I stopped to complete it.
In my case it looks like the following:
.gcloudignore (Click to expand)
.gcloudignore (Click to expand)
# This file specifies files that are *not* uploaded to Google Cloud Platform
# using gcloud. It follows the same syntax as .gitignore, with the addition of
# "#!include" directives (which insert the entries of the given .gitignore-style
# file at that point).
#
# For more information, run:
# $ gcloud topic gcloudignore
#
.gcloudignore
# If you would like to upload your .git directory, .gitignore file or files
# from your .gitignore file, remove the corresponding line
# below:
.git
.gitignore
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib
# Test binary, build with `go test -c`
*.test
# Output of the go coverage tool, specifically when used with LiteIDE
*.out
## Added by Chloe on February 11th 2021
# go build appli
appli
# Packages that are stored locally
pkg/
src/
# Credentials and token files
*credentials.json
*token.json
Finally I can deploy using gcloud app deploy.
After a first fail due to "Unable to retrieve P4SA: <service> from GAIA. Could be GAIA propagation delay or request from deleted apps.", I openned the Google Cloud console and tried again. This time I got an "Access Not Configured" error, with a link to enable billing for the Cloud Build API. Which I did and retried, and it worked !
Deploy my Calendar App
I spent hours trying to understand why, when deploying my calendar app, I got errors related to not finding packages: context, oauth2 api/calendar.
The solution is to use a module file to manage dependencies.
The first step is to run go mod init example.com/m.
On the next build (go build appli.go), the file "go.mod", that was created in the first step, is updated with the needed dependencies.
In this process I also started using the appengine package instead of context (to install run go get google.golang.org/appengine).
So my "go.mod" file looks like:
module example.com/m
go 1.15
require (
golang.org/x/oauth2 v0.0.0-20210210192628-66670185b0cd // indirect
google.golang.org/appengine v1.6.7 // indirect
)
NOTE that I am not sure about "example.com/m" which should probably be replaced with something that make more sense.
Connect with oauth2
The connection with oauth2 has to be changed since, when deployed, I can not access the command line to enter the token. I use http pages to display messages with appengine, get the authentication code and redirect to the main page.
I followed the process described for Authenticating as an end user, using the example given in the overview. Make sure to define the scope of your oauth configuration correctly, using the Scopes for Google APIs.
I got the authentication code by reading the url query values returned after the authentication to google.
The oauth protocol is described in the page Using OAuth 2.0 to Access Google APIs. It helped me understand the whole process.
I modified step by step the calendar API quickstart example to display messages in html pages instead of command line.
The main modification to the code base were as follow:
In the handler function (Click to expand), I first try to get the OAuth2 authentication link, which will be displayed (see below).
After following the link and authenticating, the app user will be automatically redirected to the handler with an authentication code in the http request, which will allow to connect to the calendar API and start the service.
Once the service is started, I can access the events as I did before with the function refreshOccupiedDaysList().
handler function (Click to expand), I first try to get the OAuth2 authentication link, which will be displayed (see below).
After following the link and authenticating, the app user will be automatically redirected to the handler with an authentication code in the http request, which will allow to connect to the calendar API and start the service.
Once the service is started, I can access the events as I did before with the function refreshOccupiedDaysList().
func handler(w http.ResponseWriter, r *http.Request) {
if !tokenRequested{
getOAuth2Link(w,r)
return
}
if !calendarServiceStarted{
startCalendarService(w,r)
return
}
if calendarService == nil{
log.Fatal("Calendar service was not initialized properly.")
}
OccupiedDaysList := make(map[int][]OccupiedDay)
events:= []string{"event1", "event2"}
if (ctx != nil){
OccupiedDaysList, events = refreshOccupiedDaysList()
} else{
log.Fatal("The background context was not initialized properly")
}
getOAuth2Link() function (Click to expand)
getOAuth2Link() function (Click to expand)
func getOAuth2Link(w http.ResponseWriter, r *http.Request) {
redirectURL := os.Getenv("OAUTH2_CALLBACK")
if redirectURL == "" {
//redirectURL = "https://indivision-toulon.ew.r.appspot.com/"
redirectURL = "http://localhost:8080/"
// note that the redirect url has to change depending on the environment (local test or appengine)
}
oauth2Config = &oauth2.Config{
ClientID: "(redacted).apps.googleusercontent.com",
ClientSecret: "(redacted)",
RedirectURL: redirectURL,
Scopes: []string{"https://www.googleapis.com/auth/calendar.events.readonly"},
Endpoint: google.Endpoint,
}
authURL := oauth2Config.AuthCodeURL("state-token", oauth2.AccessTypeOffline)
fmt.Fprint(w, "Go to the following link and sign in to your account: \n", authURL)
tokenRequested = true
}
startCalendarService() function (Click to expand)
startCalendarService() function (Click to expand)
func startCalendarService(w http.ResponseWriter, r *http.Request){
fmt.Println("Start Calendar Service")
if r.FormValue("code") == "" {
fmt.Fprint(w, "Please reload the page and follow the link provided")
tokenRequested = false
return
}
var err error
var tok *oauth2.Token
fmt.Println("About to exchange authorization code for token")
if ctx ==nil{
fmt.Println("context is nil!")
}
authCode := r.FormValue("code")
tok, err = oauth2Config.Exchange(ctx, authCode)
if err != nil {
fmt.Fprint(w, "Error with authorization code exchange : " +err.Error())
fmt.Fprint(w, "\nAuthorization code is : "+authCode)
fmt.Fprint(w, "Please reload the page and follow the link provided")
tokenRequested = false
return
}
fmt.Println("About to create client")
client:= oauth2Config.Client(ctx, tok)
fmt.Println("Client created")
calendarService, err = calendar.New(client)
if err != nil {
log.Fatalf("Unable to retrieve Calendar client: %v", err)
fmt.Fprint(w, "error getting calendar")
}
fmt.Println("Calendar service started")
calendarServiceStarted = true
http.Redirect(w, r, "/", http.StatusSeeOther)
}
Use local oauth authentication for tests
I extracted startCalendarService() and getOAuth2Link() to two files: one for local tests and one for app engine.
I added a isLocal bool that is defined in each file, so that I can switch since the call to start the calendar service is a bit different in each case.
I now either call go run appli.go startLocalCalendarService.go or go run appli.go startAppEngineCalendarService.go.
To deploy to Google Cloud, I added startLocalCalendarService.go to .gcloudignore.
I do not use build constraints as I intented in the first place.
No credentials in git
I also extracted the ClientId and ClientSecret from the OAuth2 configuration (previously "redacted"), to a file that is now ignored by git so that there is no risk of accidentally uploading the credentials to GitHub.
Check which files are deployed
In order to check which files are deployed when running gcloud app deploy, you can go to the google cloud console to "App Engine" > "Versions" > "Tools" (in "Diagnose" column) > "Debug".
Security
In order to avoid that my app holds instances of the connection to some user calendars, I need to remove global variables. As mentionned in the configuration doc, my "app should be "stateless" so that nothing is stored on the instance".
I created a cookie with a random number that maps each user to their token (for the time of the session).
It is transmitted through the http request.
The Path property of the cookie must be set to Path: "/" given that I am using redirections (otherwise the coookie persists only for the same url, see this page).
Cookie creation in the callback from authentication function (Click to expand).
The tokenCookies map allows to retrieve the token later on (see below).
Cookie creation in the callback from authentication function (Click to expand).
The tokenCookies map allows to retrieve the token later on (see below).
authCode := r.FormValue("code")
ctx := appengine.NewContext(r)
cookieId := strconv.FormatInt(rand.Int63(),10)
cookie := http.Cookie{Name : "CookieId",
Value : cookieId,
Path: "/"}
http.SetCookie(w, &cookie)
tokenCookies[cookieId], err = oauth2Config.Exchange(ctx, authCode)
Retrieve cookie to restart the calendar service on refresh (Click to expand).
The tokenCookies map allows to retrieve the token saved in the authentication callback.
Retrieve cookie to restart the calendar service on refresh (Click to expand).
The tokenCookies map allows to retrieve the token saved in the authentication callback.
func getTokenFromCookie(w http.ResponseWriter, r *http.Request) *oauth2.Token {
cookie, err := r.Cookie("CookieId")
if err != nil{
fmt.Println("Could not retrieve the cookie: "+err.Error())
return nil
} else{
cookieId:= cookie.Value
return tokenCookies[cookieId];
}
}
It is good practice to use constants and not variables in the code. Maps however cannot be made constant for various reasons. I followed this page that shows how to use an initializer function instead.
Avoid Cross Site Request Forgery
Cross Site Request Forgery (CSRF) can be avoided by using the "state" parameter of the cookie, see description of Oauth2 authorization framework. I initialize the state parameter with a random value, and change it everytime a connection is made. This is probably not viable for a large application with concurrent authentication, but I assume that for my application there won't be 2 persons trying to connect at the exact same time, and if they do the session will restart.
Use a random string for the state parameter (Click to expand).
Use a random string for the state parameter (Click to expand).
var stateString string;
func initializeStateString(){
stateString = strconv.FormatInt(rand.Int63(),10)
}
// in main
initializeStateString()
// getting the link to authenticate
authURL := oauth2Config.AuthCodeURL(stateString, oauth2.AccessTypeOffline)
// retrieving the authentication code to connect
if r.FormValue("state") != stateString{
log.Fatalf("The state is invalid, closing the session")
} else{
initializeStateString() //update the state parameter for the next authentication
}
authCode := r.FormValue("code")
...
Add favicon handler
faviconHandler (Click to expand) in needed, otherwise when the browser asks for the favicon, the regular handler is called.
faviconHandler (Click to expand) in needed, otherwise when the browser asks for the favicon, the regular handler is called.
func faviconHandler(w http.ResponseWriter, r *http.Request) {
http.ServeFile(w, r, "favicon.ico")
}
http.HandleFunc("/favicon.ico", faviconHandler)
Avoid reconnecting everyday
In order to avoid reauthentication for every session, I decided to store tokens in cookies. This is to be avoided! Only the access token should be stored in cookies, the refresh token should be stored in a database. Given that I don't want a database for the few users I will have (literally 3 or 4), I store the token in cookies. If my user access the app from a common computer, anyone will be able to access the calendar events of the authenticated user, which is wrong!
This approach did work in localhost when adding the AuthCodeOption AccessTypeOffline to the exchange of the authentication code for a token. However, when deployed, this is not a viable option, since the refresh token is only sent once on the first permission authorisation for the API. It is resend if we go to our google account and revoke the authorisation. By design, I thus need to store the refresh token on my server, which will be better in terms of security so yay !
I use file storage on the cloud (upload and download) to save my refresh token.
In order to run the app locally I need to register credential locally as described here and the bucket name (which I define in app.yaml for appengine) where the files are stored. The bucket is created automatically by appengine.
export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/<my-key>.json"
export BUCKET_NAME="<project-name>.appspot.com"
For some reason (potentially a security breach?) I can use the same refresh token for multiple accounts.