I’m going to build a twitter clone using NodeJS, but without any dependencies.
I’ve written many Node applications using dependencies like express, jest, lodash, webpack, typescript, typeorm and many more. My applications can have tens of these dependencies, and some of these dependencies will also have many dependencies, and I end up with literally 1000s of dependencies.
Updating these projects can be a major pain as dependencies get outdated and need updating, and I got tired of it. I decided to challenge myself to write an application with not a single dependency.
In this blog, I’ll start with a simple HTTP server and some tests.
Normally, I use express as a webserver, and I use jest for testing. Additionally, for testing HTTP requests, I like to use supertest and jsdom for checking the HTML responses.
Naturally, without dependencies, I can’t use any of these things.
Before I start working on an actual HTTP server, let’s consider how to test the application. I can start an HTTP server for testing and use Node’s fetch or http client to query the server, but this makes testing slow and dependent on the network stack.
So instead, I’m going to abstract away any HTTP details. I’m going to define a boundary that accepts plain objects representing HTTP requests and returns plain objects representing HTTP responses.
// request
{
method: "GET",
path: "/profile/1234",
cookies: { "user-id": 54 },
}
// response
{
status: 200,
headers: { "Content-Type": "text/html" },
content: "<!DOCTYPE ..."
}
That way I can convert a http request as implemented by NodeJS to the plain object representing the request, pass it to the implementation behind the boundary, and then write the response to the NodeJS http response.
Meanwhile, in the tests, I can just mock the requests by creating object literals representing the requests and do assertions on the response object.
Checking HTTP responses where an application would return a HTML response could be a bit of a pain. For example, let’s assert the page returns an H1 containing “hello world”:
expect(response.content).toContain("<h1>Hello world</h1>");
For the following responses, the assertion would fail:
<h1>
Hello world
</h1>
<h1 class="my-header">Hello world</h1>
<h2 class="h1">Hello world</h2>
Meanwhile, I usually don’t care the exact formatting or semantics of the HTML document. I just want to know if some content is or isn’t rendered. Easy, you might think:
expect(response.content).toContain("Hello world");
This causes all the tests to pass. Great. But what if the page doesn’t render a ‘Hello world’? Well…
FAILED:
"Hello world"not found in
<main id="content" class="mw-body" role="main"> <header class="mw-body-header vector-page-titlebar"> <nav role="navigation" aria-label="Contents" class="vector-toc-landmark"> <div id="vector-page-titlebar-toc" class="vector-dropdown vector-page-titlebar-toc vector-button-flush-left"> <input type="checkbox" id="vector-page-titlebar-toc-checkbox" role="button" aria-haspopup="true" data-event-name="ui.dropdown-vector-page-titlebar-toc" class="vector-dropdown-checkbox " aria-label="Toggle the table of contents"> <label id="vector-page-titlebar-toc-label" for="vector-page-titlebar-toc-checkbox" class="vector-dropdown-label cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only" aria-hidden="true"><span class="vector-icon mw-ui-icon-listBullet mw-ui-icon-wikimedia-listBullet"></span> <span class="vector-dropdown-label-text">Toggle the table of contents</span> </label> <div class="vector-dropdown-content"> <div id="vector-page-titlebar-toc-unpinned-container" class="vector-unpinned-container"> <div id="vector-toc" class="vector-toc vector-pinnable-element"> <div class="vector-pinnable-header vector-toc-pinnable-header vector-pinnable-header-pinned" data-feature-name="toc-pinned" data-pinnable-element-id="vector-toc"> <h2 class="vector-pinnable-header-label">Contents</h2> <button class="vector-pinnable-header-toggle-button vector-pinnable-header-pin-button" data-event-name="pinnable-header.vector-toc.pin">move to sidebar</button> <button class="vector-pinnable-header-toggle-button vector-pinnable-header-unpin-button" data-event-name="pinnable-header.vector-toc.unpin">hide</button> </div> <ul class="vector-toc-contents" id="mw-panel-toc-list"> <li id="toc-mw-content-text" class="vector-toc-list-item vector-toc-level-1 vector-toc-level-1-active vector-toc-list-item-active"> <a href="#" class="vector-toc-link"> <div class="vector-toc-text">(Top)</div> </a> </li> <li id="toc-History" class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded"> <a class="vector-toc-link" href="#History"> <div class="vector-toc-text"> <span class="vector-toc-numb">1</span>History</div> </a> <button aria-controls="toc-History-sublist" class="cdx-button cdx-button--weight-quiet cdx-button--icon-only vector-toc-toggle" aria-expanded="true"> <span class="vector-icon vector-icon--x-small mw-ui-icon-wikimedia-expand"></span> <span>Toggle History subsection</span> </button> <ul id="toc-History-sublist" class="vector-toc-list"> <li id="toc-Origin" class="vector-toc-list-item vector-toc-level-2"> <a class="vector-toc-link" href="#Origin"> <div class="vector-toc-text"> <span class="vector-toc-numb">1.1</span>Origin</div> </a> <ul id="toc-Origin-sublist" class="vector-toc-list"> <li id="toc-Simultaneous_references" class="vector-toc-list-item vector-toc-level-3"> <a class="vector-toc-link" href="#Simultaneous_references"> <div class="vector-toc-text"> <span class="vector-toc-numb">1.1.1</span>Simultaneous references</div> </a> <ul id="toc-Simultaneous_references-sublist" class="vector-toc-list"> </ul> </li> </ul> </li> <li id="toc-Growth_in_2008" class="vector-toc-list-item vector-toc-level-2"> <a class="vector-toc-link" href="#Growth_in_2008"> <div class="vector-toc-text"> <span class="vector-toc-numb">1.2</span>Growth in 2008</div> </a> <ul id="toc-Growth_in_2008-sublist" class="vector-toc-list"> </ul> </li> <li id="toc-Later_usage" class="vector-toc-list-item vector-toc-level-2"> <a class="vector-toc-link" href="#Later_usage"> <div class="vector-toc-text"> <span class="vector-toc-numb">1.3</span>Later usage</div> </a> <ul id="toc-Later_usage-sublist" class="vector-toc-list"> </ul> </li> </ul> </li> <li id="toc-Reaction" class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded"> <a class="vector-toc-link" href="#Reaction"> <div class="vector-toc-text"> <span class="vector-toc-numb">2</span>Reaction</div> </a> <ul id="toc-Reaction-sublist" class="vector-toc-list"> </ul> </li> <li id="toc-See_also" class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded"> <a class="vector-toc-link" href="#See_also"> <div class="vector-toc-text"> <span class="vector-toc-numb">3</span>See also</div> </a> <ul id="toc-See_also-sublist" class="vector-toc-list"> </ul> </li> <li id="toc-References" class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded"> <a class="vector-toc-link" href="#References"> <div class="vector-toc-text"> <span class="vector-toc-numb">4</span>References</div> </a>
There’s no fun in debugging that response.
There are other options.
I can make the server only respond with JSON and handle the rendering on the client using a React app. That way, I can just test the JSON responses, which are much easier to debug and reason about.
However, I don’t want to introduce a whole new app just to make testing the server easier. Single Page Applications can quickly become very complex and are prone to errors. In my experience, Server-Side Rendering applications are easier to build and maintain, less prone to errors and can be just as responsive.
In server-side generated webapplications, you usually use templates to convert data to html. So what if I make the boundary just return a template name and all data required to render the page using that template?
// request
{
method: "GET",
path: "/profile/1234",
cookies: { "user-id": 54 },
}
// response
{
status: 200,
template: "public-profile-show",
variables: {
user: {
id: 54,
name: "John Doe",
},
posts: [
{ id: 55412, message: "Have you seen the new iThing?",
createdAt: 1699788972 }
]
}
}
Then in my test, I can just assert the template name and data, which is much easier to reason about and results in much more accurate failure messages.
As long as I don’t too much crazy stuff in the templates, I can generally assume that if the data and template returned are correct and the template renders without error, the page should pass the test.
This method of testing isn’t perfect. It’s possible for a template to just render nothing at all, and the tests would still pass. In my experience, there’s just no way to test whether a HTML document actually renders some content. Even if the content would be present in the HTML document, it could be rendered incorrectly or not at all using CSS.
expect(response.content).toContain("Hello world");
<!-- PASS! -->
<h1 style="display: none;">Hello world!</h1>
<!-- LGTM! -->
<h1 style="position: absolute; top: -1000;">Hello world!</h1>
<!-- SURE! -->
<h1 style="color: transparent;">Hello world!</h1>
<!-- PERFECT! -->
<h1 style="font-size: 0px;">Hello world!</h1>
<!-- YEAH! -->
<h1 id="foo">Hello world!</h1>
<script>document.querySelector("#foo").innerHTML = "";</script>
Therefore, in my opinion, templates can only be tested by a pair of human eyes, rendering them using a real browser. Ideally, this manual testing can be done in multiple browsers very quickly.
To test the templates, I’ll could render the page in every test and save the output. If there’s a fatal error rendering the page, the test will fail. If the rendering succeeds, I can manually check the HTML document by opening it in my browser. Maybe I’ll create an overview page with all HTML documents in iframes to quickly review multiple pages.
After implementing some example pages and building an .ejs-like template language, the code and tests look like this:
class App {
async handleRequest(req) {
const fn = this[`${req.method} ${req.path}`];
if (!fn)
return render("not-found.ejs", req, { status: 404 });
return fn.call(this, req);
};
async "GET /login"() {
return render("login.ejs", { errorCode: null, params: { username: "" } });
}
async "POST /login"({ body }) {
const { username, password } = body;
const { user, error } = await this.#auth.login(username, password);
if (user)
return redirect("/profile", { cookies: { username: user.name } });
return render("login.ejs", {
errorCode: error.description,
params: { username }
});
}
}
The render and redirect functions just return response objects.
function render(template, variables = {}, overrides = {}) {
return { template, variables, status: 200, ...overrides };
}
function redirect(location, overrides = {}) {
return { location, status: 302, ...overrides };
}
They’re very simple one-liners meant to improve readability and set some defaults.
describe("GET /login", () => {
it("renders login template", async () => {
await GET("/login");
assertRender("login.ejs", { params: { username: "" }, errorCode: null });
});
});
describe("POST /login", () => {
it("with unknown username param, returns user not found error", async () => {
await POST("/login", { username: "foo", password: "bar" });
assertRender("login.ejs", {
params: { username: "foo", },
errorCode: "USER_NOT_FOUND",
});
});
it("with valid params, sets cookie and redirects to profile", async () => {
await auth.signup("foo", "bar");
await POST("/login", { username: "foo", password: "bar" });
assertRedirect("/profile", { cookies: { username: "foo" } });
});
});
In a beforeEach function, the app is initialized. In my testing file, I have POST and GET functions available. Even though they’re oneliners, they improve readability by reducing noise.
async function POST(path, body) {
res = await app.handleRequest({ method: "POST", path, body });
}
async function GET(path) {
res = await app.handleRequest({ method: "GET", path });
}
assertRender and assertRedirect check the response in res, assigned by GET or POST. assertRender also renders the template, to check for errors in the template.
function assertRender(template, variables = {}, overrides = {}) {
assert.deepStrictEqual(res,
{ status: 200, template, variables, ...overrides });
if (res.template)
renderTemplate(res.template, res.variables);
}
function assertRedirect(location, overrides = {}) {
assert.deepStrictEqual(res, { status: 302, location, ...overrides });
}
renderTemplate is created by my template engine. It is a function that accepts a template name and variables and returns HTML as a string.
This template engine renders an EJS-like template like this:
<h1>Login!</h1>
<form method="POST" action="/login">
<% if (errorCode) { %>
<p><%= {
"USER_NOT_FOUND": "User not found",
"INVALID_PASSWORD": "Invalid password"
}[errorCode] ?? errorCode %></p>
<% } %>
<label>
<strong>Username</strong><br>
<input type="text" name="username" value="<%= params.username %>">
</label><br>
<label>
<strong>Password</strong><br>
<input type="password" name="password">
</label><br>
<button type="submit">Login</button>
</form>
To some nice HTML.
In the next blog, we’ll do a deep dive into this template language.
As a bonus, here is the code responsible for actually processing HTTP requests and generating responses.
const server = http.createServer(async (req, res) => {
try {
const url = new URL(req.url, `http://${req.headers.host}`);
const body = await new Promise((resolve) => {
if (req.method != "POST") {
resolve(null);
return;
}
const bodyData = [];
req.on("data", (chunk) => bodyData.push(chunk));
req.on("end", () => { resolve(querystring.parse(bodyData.join(""))); });
});
let { status, json, headers, template, variables, cookies, location } = await app.handleRequest({
method: req.method,
path: url.pathname,
body,
});
if (status) res.statusCode = status;
if (headers)
for (const key in headers)
if (headers.hasOwnProperty(key))
res.setHeader(key, value);
if (cookies)
for (const key in cookies)
if (cookies.hasOwnProperty(key)) {
const value = [
`${key}=${cookies[key] ?? ""}`,
`Path=/`,
`HttpOnly`,
`SameSite=Strict`,
cookies[key] ? `Max-Age=${YEAR_SECONDS}` : `Max-Age=-1`
].join("; ");
res.setHeader("set-cookie", value);
}
if (location) {
res.setHeader("Location", location);
res.end();
return;
} else if (json) {
res.setHeader("content-type", "application/json");
res.end(JSON.stringify(json));
return;
} else if (template) {
res.setHeader("content-type", "text/html");
res.end(render(template, variables));
return;
}
throw new Error(`Request handler had no response`);
} catch (error) {
res.statusCode = 500;
res.setHeader("content-type", "text/plain");
res.end(`SOMETHING WENT WRONG\n${error.stack}`);
}
});