Accessing GitHub's REST API with CURL
REST APIs are popular. First mentioned in Roy Fielding’s dissertation
it describes an architecture based on the World Wide Web. REST APIs
expose resources. The main prerequisites of good REST APIs are
- Addressability Every resource is addressable via an uniform
resource identifier. - Uniform Interface HTTP is used as an interface to the manipulated
the various resources the API exposes. - Interconnectednes Just like website are linked together,
resources should provide links to related resources. - Statelessness The server should be kept stateless. Any state that
is necessary should be passed along the request.
When discussing RESTful APIs it good to have some real world
examples. REST APIs that are accessible via the internet often shield
their interface behind some form of authentication. This adds a
barrier to using these API as demonstration.
In this blog post we will show how to use GitHub REST API, that needs
authentication, from the command line using curl.
1 GitHub
GitHub is a hosting service for git repositories. GitHub’s logo sports
the sub-title “social coding” and it can be seen as a social network
for developers.
The GitHub’s REST API documentation can be found at
https://developer.github.com/v3/.
2 EndPoints
We can request all the endpoints that GitHub exposes with the
following command
curl -X GET https://api.github.com
{
"current_user_url": "https://api.github.com/user",
"authorizations_url": "https://api.github.com/authorizations",
"code_search_url": "https://api.github.com/search/code?q={query}{&page,per_page,sort,order}",
"emails_url": "https://api.github.com/user/emails",
"emojis_url": "https://api.github.com/emojis",
"events_url": "https://api.github.com/events",
"feeds_url": "https://api.github.com/feeds",
"following_url": "https://api.github.com/user/following{/target}",
"gists_url": "https://api.github.com/gists{/gist_id}",
"hub_url": "https://api.github.com/hub",
"issue_search_url": "https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}",
"issues_url": "https://api.github.com/issues",
"keys_url": "https://api.github.com/user/keys",
"notifications_url": "https://api.github.com/notifications",
"organization_repositories_url": "https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}",
"organization_url": "https://api.github.com/orgs/{org}",
"public_gists_url": "https://api.github.com/gists/public",
"rate_limit_url": "https://api.github.com/rate_limit",
"repository_url": "https://api.github.com/repos/{owner}/{repo}",
"repository_search_url": "https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}",
"current_user_repositories_url": "https://api.github.com/user/repos{?type,page,per_page,sort}",
"starred_url": "https://api.github.com/user/starred{/owner}{/repo}",
"starred_gists_url": "https://api.github.com/gists/starred",
"team_url": "https://api.github.com/teams",
"user_url": "https://api.github.com/users/{user}",
"user_organizations_url": "https://api.github.com/user/orgs",
"user_repositories_url": "https://api.github.com/users/{user}/repos{?type,page,per_page,sort}",
"user_search_url": "https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"
}
It returns a JSON object with endpoints that can be requested.
3 Authentication
Let’s request the current user. From the endpoints listed in the
preceding section we know that we have to request https://api.github.com/user
curl -X GET https://api.github.com/user
{
"message": "Requires authentication",
"documentation_url": "https://developer.github.com/v3"
}
Unfortunatly the server sends back a status code of 401 Unauthorized
with the preceding message.
3.1 Basic Authentication
One way of authenticating is via basic authentication as specified in
RFC2617. In this form of authentication you provide the GitHub
username and password. Using curl one can pass in the -u or --user
option, providing the corresponding password with each request.
The following excerpt shows a session demonstrating this option.
> curl -X GET -u dvberkel https://api.github.com/user/dvberkel Enter host password for user 'dvberkel':
This option soon becomes tiresome, especially if you have a password
with high entropy. To mitigate the burden one could create environment
variables and use them instead.
For example, if you create the following environment variables
export GITHUB_USER=dvberkel export GITHUB_PASSWORD=abcd1234
A request could be made with the command
curl -X GET -u $GITHUB_USER:$GITHUB_PASSWORD https://api.github.com/user
With the result
{
"login": "dvberkel",
"id": 493347,
"avatar_url": "https://avatars.githubusercontent.com/u/493347?v=3",
"gravatar_id": "",
"url": "https://api.github.com/users/dvberkel",
"html_url": "https://github.com/dvberkel",
"followers_url": "https://api.github.com/users/dvberkel/followers",
"following_url": "https://api.github.com/users/dvberkel/following{/other_user}",
"gists_url": "https://api.github.com/users/dvberkel/gists{/gist_id}",
"starred_url": "https://api.github.com/users/dvberkel/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/dvberkel/subscriptions",
"organizations_url": "https://api.github.com/users/dvberkel/orgs",
"repos_url": "https://api.github.com/users/dvberkel/repos",
"events_url": "https://api.github.com/users/dvberkel/events{/privacy}",
"received_events_url": "https://api.github.com/users/dvberkel/received_events",
"type": "User",
"site_admin": false,
"name": "Daan van Berkel",
"company": null,
"blog": "http://dvberkel.github.com",
"location": null,
"email": "daan.v.berkel.1980@gmail.com",
"hireable": false,
"bio": null,
"public_repos": 221,
"public_gists": 25,
"followers": 22,
"following": 63,
"created_at": "2010-11-23T12:41:08Z",
"updated_at": "2014-11-24T10:13:01Z",
"private_gists": 0,
"total_private_repos": 3,
"owned_private_repos": 2,
"disk_usage": 225912,
"collaborators": 1,
"plan": {
"name": "small",
"space": 1228800,
"collaborators": 0,
"private_repos": 10
}
}
3.2 2-Factor Authentication
GitHub supports 2-Factor Authentication which provides additional
security. If a user uses 2-factor authentication basic authentication
is a bit more complicated. When firing a request
curl -X GET -u $GITHUB_USER:$GITHUB_PASSWORD https://api.github.com/user
The server responds with
{
"message": "Must specify two-factor authentication OTP code.",
"documentation_url": "https://developer.github.com/v3/auth#working-with-two-factor-authentication"
}
Furthermore, it sends back a X-GitHub-OTP: required; app or
X-GitHub-OTP: require; sms header, indicating that a one-time
password is required. These one-time passwords are acquired via an
app, such as Google Authenticator, or via SMS.
Once required, the one-time password needs to be send via the
X-GitHub-OTP header as shown in the code below.
curl -X GET -u $GITHUB_USER:$GITHUB_PASSWORD -H 'X-GitHub-OTP: 123456' https://api.github.com/user
3.3 Access Token
Providing a username and password with every request soon becomes
tiresome. Especially for users that have enabled 2-factor
authentication.
Although the burden is somewhat alliviated by introducing environment
variables, this introduces the possibillity that someone gleans your
password by taking control of your computer and echoing these
variables.
> echo $GITHUB_PASSWORD
abcd1234
GitHub offers other means of authentication. One of them are access
tokens. Access tokens are very well suited for our use case,
i.e. accessing the GitHub REST API from the command line.
An access token can be easily created and revoked. It offers
fine-grained access controlled via scopes.
3.3.1 Web Interface
One way to create access tokens is via the web interface that GitHub
offers. You access it by clicking the settings icon.
Next you have to select Applications
and create a new token
You can give the access token a meaningfull description so you can
differentiate it in the list of all access tokens.
Furthermore you can selectively pick scopes to control what this token
gives access to.
When you finally click Generate token
your access token will be shown.
For security reasons this is the only time you the access token is
shown. The next time you visit the page it will only show the
description.
You should treat access tokens as if they are passwords. Do not share
them easily and make sure to remove unused access tokens often.
3.3.2 REST API
An other way of creating access tokens is via the Authorizations
API. You access the Authorization API with basic authentication as
explained above. For example the code below generates an access token
to the users email address.
curl -X POST -u $GITHUB_USER:$GITHUB_PASSWORD -H 'Content-Type: application/json' -d '{"scopes": ["user:email"],"note": "blog example"}' https://api.github.com/authorizations
We send along our credentials and data. The data is send as JSON so we
add an appropriate HTTP header. The result send back includes our
access token.
{
"id": 12345678,
"url": "https://api.github.com/authorizations/12345678",
"app": {
"name": "blog example (API)",
"url": "https://developer.github.com/v3/oauth_authorizations/",
"client_id": "00000000000000000000"
},
"token": "1234567890abcdef1234567890abcdef12345678",
"note": "blog example",
"note_url": null,
"created_at": "2014-11-25T11:50:17Z",
"updated_at": "2014-11-25T11:50:17Z",
"scopes": [
"user:email"
]
}
Just like our username and password, the token can be stored in an
environment variable for ease access
export GITHUB_TOKEN=1234567890abcdef1234567890abcdef12345678
When we are done with the token we can use the API to delete the token
as well.
curl -X DELETE -u $GITHUB_USER:$GITHUB_PASSWORD https://api.github.com/authorizations/12345678
3.4 Using Acces Token
With our access token nicely tucked away in our environment variable
GITHUB_TOKEN we can start using it. We will use the access token to
retrieve the user information.
Instead of passing along the username and password we pass along our
access token and a password of x-oauth-basic.
curl -X GET -u $GITHUB_TOKEN:x-oauth-basic 'https://api.github.com/user'
{
"login": "dvberkel",
"id": 493347,
"avatar_url": "https://avatars.githubusercontent.com/u/493347?v=3",
"gravatar_id": "",
"url": "https://api.github.com/users/dvberkel",
"html_url": "https://github.com/dvberkel",
"followers_url": "https://api.github.com/users/dvberkel/followers",
"following_url": "https://api.github.com/users/dvberkel/following{/other_user}",
"gists_url": "https://api.github.com/users/dvberkel/gists{/gist_id}",
"starred_url": "https://api.github.com/users/dvberkel/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/dvberkel/subscriptions",
"organizations_url": "https://api.github.com/users/dvberkel/orgs",
"repos_url": "https://api.github.com/users/dvberkel/repos",
"events_url": "https://api.github.com/users/dvberkel/events{/privacy}",
"received_events_url": "https://api.github.com/users/dvberkel/received_events",
"type": "User",
"site_admin": false,
"name": "Daan van Berkel",
"company": null,
"blog": "http://dvberkel.github.com",
"location": null,
"email": "daan.v.berkel.1980@gmail.com",
"hireable": false,
"bio": null,
"public_repos": 221,
"public_gists": 25,
"followers": 22,
"following": 63,
"created_at": "2010-11-23T12:41:08Z",
"updated_at": "2014-11-25T08:12:06Z",
"private_gists": 0,
"total_private_repos": 3,
"owned_private_repos": 2,
"disk_usage": 225912,
"collaborators": 1,
"plan": {
"name": "small",
"space": 1228800,
"collaborators": 0,
"private_repos": 10
}
}
4 Conclusion
Demonstrating GitHub’s REST API from the command line is fairly
easy. By creating an access token and storing it in an environment
variable curl can be used to access the resources that GitHub exposes.