HTTP and curl

https://github.com/heig-vd-dai-course

Web · PDF

L. Delafontaine and H. Louis, with the help of GitHub Copilot.

Based on the original course by O. Liechti and J. Ehrensberger.

This work is licensed under the CC BY-SA 4.0 license.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

Objectives

  • Understand the basics of HTTP
  • Understand the basics of APIs
  • Learn how to use curl
  • Learn how to design and document a simple API
  • Learn how to develop a simple API
  • Learn how to use a simple API
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

Disclaimer

More details for this section in the course material. You can find other resources and alternatives as well.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

Disclaimer

  • This is not a course on web development
  • Many many things are not covered
  • Focus on HTTP version 1.1
  • Javalin used for learning purposes
  • For production, use a framework like Quarkus or Spring Boot
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

Prepare and setup your environment

More details for this section in the course material. You can find other resources and alternatives as well.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

curl

  • An open source command-line tool
  • Used to transfer data using various protocols
    • HTTP/HTTPS
    • FTP
    • etc.
  • Used to test APIs
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

Javalin

  • A lightweight web framework for Java and Kotlin
  • Easy to learn and use: perfect for learning purposes
  • Production ready but not as powerful as Quarkus or Spring Boot
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

HTTP

More details for this section in the course material. You can find other resources and alternatives as well.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

HTTP

  • Initiated by Tim Berners-Lee at CERN in 1989
  • First release in 1990 to transfer HyperText Markup Language (HTML) documents
  • Built on top of TCP (HTTP/1.0, HTTP/1.1 and HTTP/2.0) or UDP/QUIC (HTTP/3)
  • Ports 80 (HTTP) or 443 (HTTPS)
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0
  • Hyper Text Transfer Protocol (HTTP) based on TCP
  • Application layer protocol with many features
  • Used to transfer data between a client (an user agent) and a server
  • A client can be a web browser, a mobile application, a command-line tool, household appliance, etc.
  • The client requests a resource from the server
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

HTTP versions

Multiple versions exist:

  • HTTP/1.0 (1996)
  • HTTP/1.1 (1997)
  • HTTP/2 (2015)
  • HTTP/3 (2022)

The most used version is HTTP/1.1. Each version is to improve performance.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

HTTP resources

  • A resource is identified by a Uniform Resource Locator (URL)
  • A resource can be a file, a document, a video, etc.
  • Sometimes called an endpoint or a route
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

An example of a URL is the following:

https://gaps.heig-vd.ch/consultation/fiches/uv/uv.php?id=6573
  • Protocol: http:// or https://
  • Host: gaps.heig-vd.ch
  • Port: :80 or :443 (optional)
  • Path: /consultation/fiches/uv/uv.php
  • Query parameters: ?id=6573

This resource returns a HTML document.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

URL encoding

  • URLs can only contain a limited set of characters
  • Some characters are reserved for special purposes
  • Some characters must be encoded: (Space -> %20)
  • For example, Hello world becomes Hello%20world
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

HTTP request methods

  • GET - Get a resource
  • POST - Create a resource
  • PATCH/PUT - Update a resource
  • DELETE - Delete a resource

A browser does GET methods by default.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

HTTP request and response format

  • To request a resource, a client sends a HTTP request to a server
  • The server processes the request and sends back a HTTP response

HTTP is based on a request-response model.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

The HTTP request and response are composed of:

  • A start line with:
    • The method
    • The URL
    • The version
  • Headers with metadata
  • An empty line
  • An optional body with data
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

Structure of a HTTP request:

<HTTP method> <URL> HTTP/<HTTP version>
<HTTP headers>
<Empty line if there is a body>
<HTTP body (optional)>

Structure of a HTTP response:

HTTP/<HTTP version> <HTTP status code> <HTTP status message>
<HTTP headers>
<Empty line if there is a body>
<HTTP body>
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

A HTTP request example:

GET / HTTP/1
Host: gaps.heig-vd.ch
User-Agent: curl/8.1.2
Accept: */*

A HTTP response example:

HTTP/1.1 200 OK
Content-Type: text/html; charset=ISO-8859-1
Content-Length: 6111

<!DOCTYPE HTML>
...
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

HTTP response status codes

Grouped by categories:

  • 1xx: Informational
  • 2xx: Success
  • 3xx: Redirection
  • 4xx: Client error
  • 5xx: Server error
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

HTTP path parameters, query parameters and body

These elements are used to pass data to the server.

Path parameters and query parameters are part of the URL.

Headers are part of the HTTP request or response.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

HTTP path parameters

An example of a path parameter is the following:

/users/{user-id}

The {user-id} part is a path parameter.

With values: /users/123 -> 123 is the user ID.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

HTTP query parameters

An example of a query parameter is the following:

/users?firstName=John&lastName=Doe

The ? character separates the path from the query parameters.

The & character separates query parameters.

Each query parameter is composed of a key and a value.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

HTTP body

An example of a HTTP body is the following:

HTTP/1.1 200 OK
Content-Type: text/html; charset=ISO-8859-1
Content-Length: 6111

<!DOCTYPE HTML [...]>
<html>

[...]

</html>
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

HTTP headers

HTTP headers are used to pass metadata to/from the server.

  • Accept - The media types accepted by the client
  • Content-Type - The media type of the body
  • Content-Length - The length of the body
  • User-Agent - The user agent of the client
  • Host- The host of the server
  • Set-Cookie - The cookies set by the server
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

HTTP content negotiation

The Accept header is used to negotiate the content type between the client and the server. These are based on the MIME types:

  • Accept: text/html - HTML
  • Accept: application/json - JSON
  • etc.

The same URL can return different content types based on the Accept header.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

HTTP sessions (stateless vs. stateful)

  • Stateless: the application does not store any data between requests
  • Stateful: the application stores data between requests

HTTP is a stateless protocol.

The server does not store any data between requests.

Let's illustrate this with an example.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

Imagine that you have a website with a few pages that you can access:

  • A login page (/login - public)
  • A profile page (/profile - private)
  1. You access the login page and fill in your username and password and you click on the "Login" button.
  2. The server checks your credentials accept your login request.
  3. You try to access the profile page but the server rejects your request. Why?

The server does not know who you are..!

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

Why does the server not know who you are?

It is because you do not have a way to state who you are. You do not have a session with the server.

They are two ways to create a session:

  • Using a query parameter
  • Using cookies
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

HTTP sessions using a query parameter

A query parameter can be used to create a session:

C: POST /login
S: 302 Found (redirect to /profile?token=1234567890)
C: GET /profile?token=1234567890
S: 200 OK (profile page)

Advantages: Easy to implement.

Disadvantages: The token is visible in the URL (security issue).

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

HTTP sessions using cookies

A cookie can be used to create a session:

C: POST /login
S: 302 Found (redirect to /profile and set a cookie with the token)
C: GET /profile (the cookie is sent by the client)
S: 200 OK (profile page)

Advantages: The token is not visible in the URL (more secure).

Disadvantages: A bit more complex to implement.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

API design

More details for this section in the course material. You can find other resources and alternatives as well.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

API design

Developing a web application is not easy.

In order to make it easier, we follow patterns and a set of rules such as an Application Programming Interface (API).

An API is a contract between the client and the server that must be documented.

Most APIs are based on HTTP and exchange data in JSON format, the most used format for APIs.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

JSON is an easy to read and write format for humans. It is also easy to parse for computers.

Example of a JSON document:

{
  "firstName": "John",
  "lastName": "Doe",
  "age": 42
}
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

Simple APIs with CRUD operations

CRUD stands for:

  • Create
  • Read
  • Update
  • Delete

CRUD APIs are used to manage data.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

A simple API with CRUD operations to manage users will expose the following endpoints:

  • POST /users - Create a new user
  • GET /users - List all users
  • GET /users/{id} - Read a user
  • PUT /users/{id} - Update a user
  • PATCH /users/{id} - Partially update a user
  • DELETE /users/{id} - Delete a user
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

REST APIs

A REST API is a set of (strict) rules to design APIs.

The REST pattern is based around a six following principles.

Is is an improvement over CRUD APIs.

Not all APIs are REST APIs but all REST APIs are APIs.

REST APIs are hard to implement correctly. In this course, we will stay with CRUD APIs. We mention REST APIs for completeness.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

How to document an API

  • Documentation is important for developers as well as users
  • An API exposes the features of an application to the outside world
  • There exist many tools to document APIs such as OpenAPI
  • As these tools are complex, we will use a simple solution: a text file #Define an application protocol
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

How to persist data

  • In the course material, we store data in memory
  • Data can be (and should be!) stored in a database
  • Out of scope for this course but you can find resources in the course material
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

How to secure an API

  • Not all APIs are public
  • Some APIs are private and require authentication
  • Out of scope for this course but you can find resources in the course material.

It is more important to understand the basics of how to design, how to develop and how to document an API.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

Practical content

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

What will you do?

  • Try out all the main concepts of HTTP (methods, status codes, headers, JSON, etc.)
  • Learn how to use curl
  • Build a simple web application to manage users
  • Learn how to design and document a simple API
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

Find the practical content

You can find the practical content for this chapter on GitHub.

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

Finished? Was it easy? Was it hard?

Can you let us know what was easy and what was difficult for you during this chapter?

This will help us to improve the course and adapt the content to your needs. If we notice some difficulties, we will come back to you to help you.

➡️ GitHub Discussions

You can use reactions to express your opinion on a comment!

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

What will you do next?

In the next chapter, you will learn the following topics:

  • Web infrastructures
    • How to run and maintain web applications on the Internet?
    • How to scale web applications?
    • How to secure web applications?
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0

Sources

HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0
HEIG-VD - DAI Course 2023-2024 - CC BY-SA 4.0