Querying the Twitter API in Golang and cleaning up the response

In this post, I am going to show you how to query the Twitter API in Golang and tidy up the response output. Note that this code includes no go routines or channels for concurrency and has all the code in a single file, which is less than ideal. Tomorrows post will show you how to resolve these issues in a much more manageable code base.

The first thing we do, as always, is we import the modules we require. Then we define two struct types. As I spoke about in this article, a struct is a data type in Golang, it can also be compared to a class in an object oriented programming language. A struct gives us the ability to set specific fields for a type of data.

Here, I’ve defined a struct for a tweet and a struct for a user. Each will have a number of attributes. We will be interacting with these structs later, so it should become clear why I’ve used them.

package main

import "github.com/amit-lulla/twitterapi" //https://godoc.org/github.com/amit-lulla/twitterapi
import "fmt"
import "strings"
import "time"
import "log"
import "net/url"
import "math"

type user_details struct {
	handle string
	url string
	timezone string
	StatusesCount int64
	location string
	CreatedAt string
	AccountAge float64
	FriendsCount int
	FollowersCount int
	FavouritesCount int
}

type tweet_details struct {
	CreatedAt string
	Age float64
	FavoriteCount int
	RetweetCount int
	}

Next, let’s have a look at the main function. Here, I only make two function calls. The first, is a call to get user details and the second is a call to get the posts for that user.

func main() {
	//query API for specific handle
	user_output := searchProfile("BBCSport")
    getPosts("BBCNews")
	fmt.Println(user_output)   
	}

Next, I have a connect function, where I use my API keys to create an API session in my Golang code. We’ll be referencing this function from other functions a little later.

func connect()  *twitterapi.TwitterApi {
	twitterapi.SetConsumerKey("x")
    twitterapi.SetConsumerSecret("x")  
	api := twitterapi.NewTwitterApi("x", "x")
	return api
}

Now we get to the interesting bit. Here is the searchProfile function. This function takes the profile name argument as a string and returns a user_type struct as the function output.

So, we make a call to the connect() function in order to make an API session. We then use that API object and make a call to the GetUsersShow function. The object returned is a searchResult struct. We can then extract specific fields from that struct (as we don’t want all of them).

The date format in the response is awful it’s in this format: Fri Apr 09 12:53:54 +0000 2020. I pass this into the CalcAge function which calculates how many days ago the account was created. We’ll look at that function a bit later.

Finally, we store our output fields into our user_details struct and return it.

func searchProfile(profileName string) user_details {
	api := connect()
	searchResult, _ := api.GetUsersShow(profileName, nil)
	FavouritesCount := searchResult.FavouritesCount
	FollowersCount := searchResult.FollowersCount
	FriendsCount := searchResult.FriendsCount
	CreatedAt := searchResult.CreatedAt
	Location := searchResult.Location
	StatusesCount := searchResult.StatusesCount
	TimeZone := searchResult.TimeZone
	URL := searchResult.URL
	handle := searchResult.Name
	
	acctAge := CalcAge(CreatedAt)

	output := user_details{
		handle : handle, 
		url : URL, 
		timezone: TimeZone,
		StatusesCount: StatusesCount, 
		location: Location, 
		CreatedAt: CreatedAt, 
		AccountAge: acctAge,
		FriendsCount: FriendsCount, 
		FollowersCount: FollowersCount, 
		FavouritesCount: FavouritesCount}

	return output
}

Next, I have the getPosts function, which again takes the profileName as an input but you’ll notice that it has no return value. That’s why in the main function, I didn’t assign the output to a variable. I just call the function because I want something to happen, I don’t need to store the value.

Again, we create a new API object and make a query to it. This time we use the GetUserTimeline function. This requires a url value to be the input, so I’ve used the url.parseQuery function to be able to convert my string into an acceptable format for the function.

Again, the output is saved to a variable called searchResult. The output this time is an iterable, so we loop over it and extract the values we want. As part of this I pass the horrible CreatedDate value into a function called CalcAge, which will be a bit further down this article. This calculates how many days old the post is. That way, we can easily select a custom timeframe of tweets we’re interested in. I also pass the date into the CleanDate function which converts our horrible date format described above into YYYY-MM-DD.

In each iteration, I add the values to the tweet_details struct. In this example, I just print the values, but you can save to a database or a file or handle it however you please.

func getPosts(profileName string) {
	api := connect()
	v, _ := url.ParseQuery("screen_name="+profileName+"&count=100&include_rts=False&tweet_mode=extended")
	searchResult, _ := api.GetUserTimeline(v)

    for _, value := range searchResult {
		CreatedAt := value.CreatedAt
		FavoriteCount := value.FavoriteCount
		RetweetCount := value.RetweetCount
		Posted := CalcAge(CreatedAt)
		CreatedDate := CleanDate(CreatedAt)

		rounded := math.Floor(Posted*100)/100 //rounds number
 
        output := tweet_details{
			CreatedAt: CreatedDate,
			Age: rounded,
			FavoriteCount: FavoriteCount,
			RetweetCount: RetweetCount,
		}
        fmt.Println(output)
		}
}

Finally, let’s look at our two cleanup functions. The first, is the CleanDate function. It takes our date in the format: Fri Apr 09 12:53:54 +0000 2020 and converts it to YYYY-MM-DD through string splits and a quite ugly if statement.

func CleanDate(CreatedAt string) string {
	month := strings.Split(CreatedAt, " ")[1]
	day := strings.Split(CreatedAt, " ")[2]
	year := strings.Split(CreatedAt, " ")[5]

    if month == "Jan" {
		month = "01"
	} else if month == "Feb" {
		month = "02"
	} else if month == "Mar" {
		month = "03"
	} else if month == "Apr" {
		month = "04"
	} else if month == "May" {
		month = "05"
	} else if month == "Jun" {
		month = "06"
	} else if month == "Jul" {
		month = "07"
	} else if month == "Aug" {
		month = "08"
	} else if month == "Sep" {
		month = "09"
	} else if month == "Oct" {
		month = "10"
	} else if month == "Nov" {
		month = "11"
	} else if month == "Dec" {
		month = "12"
	} 

	full_date := year + "-" + month + "-" + day

	return full_date
}

And finally, the calcAge function, which converts the date format just as above, then casts it as a date and subtracts the date from now() to give us an age.

func CalcAge(CreatedAt string) float64 {
	month := strings.Split(CreatedAt, " ")[1]
	day := strings.Split(CreatedAt, " ")[2]
	year := strings.Split(CreatedAt, " ")[5]

    if month == "Jan" {
		month = "01"
	} else if month == "Feb" {
		month = "02"
	} else if month == "Mar" {
		month = "03"
	} else if month == "Apr" {
		month = "04"
	} else if month == "May" {
		month = "05"
	} else if month == "Jun" {
		month = "06"
	} else if month == "Jul" {
		month = "07"
	} else if month == "Aug" {
		month = "08"
	} else if month == "Sep" {
		month = "09"
	} else if month == "Oct" {
		month = "10"
	} else if month == "Nov" {
		month = "11"
	} else if month == "Dec" {
		month = "12"
	} 

	full_date := year + "-" + month + "-" + day
	
	//define parsedDate as a time data type
	var parsedDate time.Time
	
	//convert our string datatype to be of type time
    parsedDate, err := time.Parse("2006-01-02", full_date)
    if err != nil {
        log.Fatalln(err)
    }

    today := time.Now()
	
	//subtract created date from today to get age 
    age := today.Sub(parsedDate).Hours() / 24
	return age
}