fbpx
Engineering Partner Alejandro Esquivel writes about his facial recognition tool.
Engineering Partner Alejandro Esquivel writes about his facial recognition tool.

Honing the craft of machine learning by creating a facial recognition tool: My Commit Hackathon Onboarding Project

March 16, 2021 in CHOP
 

Commit’s culture of innovation is immediately apparent when you join. Every new engineer starts with a one-week internal hackathon onboarding project, or HOP. I got to plan and build any tool or application that encourages learning about any technology. Once the week is up, Engineers present to their peers internally with a demo, which gives them a chance to meet their new colleagues and discuss what they built or learned in the process.

Tell us about what you built.

I’m interested in learning more about machine learning, so I chose to build a very basic facial recognition tool that allows you to sign in to a website using your webcam. Albeit a bit tongue-in-cheek, I named the project “FacialID.”

What is FacialD?

It is a login system that allows you to sign in to a website using your computer camera, similar to Apple’s FaceID.

The Problem

Who wants to remember so many credentials and passwords to log in to websites? Wouldn’t it be nice to log in with just a click of a button? 

The Solution

FacialID uses a browser’s WebRTC API to request access to the user’s webcam and takes a picture upon pressing the Login button. It sends the snapshot to a server, which detects the faces present in the image and compares them to a database of registered users’ faces. If it finds a match, the server returns a JWT token allowing a web application to authenticate subsequent requests. 

At this point, you’re probably not satisfied with that explanation. How does one go about comparing different faces? Or, more precisely, computing the similarity between faces? Let’s do a quick run-through of my approach. I’m writing a high-level description, so please don’t hesitate to reach out if you want more technical details.

Facial Recognition

The technology of facial recognition has been around for a very long time—since the late nineteenth century. I decided to take the data scientist’s mantra “start with the simplest model” to a new level and started with one of the earliest approaches to facial recognition, which gained traction in the 1990s, dubbed “eigenfaces.” This technology relies on a concept called dimensionality reduction. Although it has a fancy name, in principle it is quite simple. In a way, we do dimensionality reduction all the time. For example, when we try to explain a complicated topic, some of the best explanations, in my opinion, explain the largest percentage of the topic in the simplest possible terms (i.e., the least amount of concepts). In a way, that’s what happens with the eigenface approach. The dimensionality reduction algorithm (in this case PCA, or principal component analysis) takes many faces and finds a way to mathematically describe as many as possible in the fewest possible terms. For those with some math background, PCA finds a lower-dimensional subspace that is closest to your data in feature space.

A visual representation of what happens after running many faces through the PCA algorithm.

A visual representation of what happens after running many faces through the PCA algorithm.

The above illustration gives a visual representation of what happens after running many faces through the PCA algorithm. It turns out that any input face can be approximated by combining a finite set of simpler faces, or eigenfaces, with different weights.

How does this help us? Mathematically, this gives us a way to assign numerical quantities (“vectors”) to each face, and we can compute the “distance” between two pairs of numerical quantities that represent the faces. We can also use this method as a crude way to detect faces in pictures by seeing if the picture has a distance very far from all the faces we have at our disposal. It is worth noting that since we can approximate any input image using less information, PCA can also be used for image compression purposes. 

One could use the eigenface approach to then use the face’s numerical representation as an input to a supervised machine learning model, which could classify a face as Face A, Face B or Face C or only My Face versus Not My Face. But these approaches would require retraining the model every time someone signs up using their picture, which complicates the ability to support many users. So we opt to simply compute the distances between faces, which doesn’t require re-training a model after obtaining the eigenfaces (an unsupervised machine learning approach, given how it relates to clustering).

Now that we’ve unravelled what’s going on under the hood, let’s restate FacialID’s actions when someone tries to log in. The webcam takes the photo and sends it to the server; then, the server scans the image to determine whether it’s a face by scanning portions of the image and using a face-detection classifier. The extracted bounding box containing the face is given a numerical representation using the PCA/eigenface approach. The distances between the input face and each of the faces of those registered in our database are computed. We assign a threshold distance that allows us to conclude when a face is close enough to another to determine whether it is the same person. We can do this heuristically; however, data-driven conclusions are optimal. For example, computing the minimum distance between pairs of unique faces in a large dataset should give you an upper bound for the threshold. There’s lots of room to play around with this value. Of course, there’s always a trade-off between convenience and security. Making this threshold smaller makes it a bit more “secure” at the expense of sometimes requiring more than one attempt to log in, depending on a few factors I will discuss in the improvements section.

 The Demo

demoimage.gif

The code is open-sourced and available in Github here.

Future Improvements

Although this approach is arguably one of the simplest ways to implement facial recognition, it is prone to many limitations. Particularly, lighting conditions directly affect the “distance” between two faces, which can affect the ability of the program to correctly log in someone in very different lighting conditions compared to the image used to sign up. Furthermore, for this approach to work best, someone needs to face the camera directly for their picture during signup and subsequent logins. Tilting your face at arbitrary angles also impacts our “distance” measurements. You may not be logged in if you’re facing the camera at an awkward angle. For instance, at 45 degrees to the left, relative to the camera. 

  1. One of the first improvements would be to play around with adaptive histogram equalization techniques to see if lighting conditions could be automatically adjusted prior to constructing an eigenface representation.

  2. Another improvement would be to ensure that facial features of all faces line up, for example, ensuring that the eyes, nose, and mouth are always at a point closest to the centre of the image, which would allow a more consistent way of comparing pairs of faces.

  3. It has been discussed in several papers that slightly blurring faces also improves PCA performance by removing “sharp” features or noisy artifacts that may be present in the inputted image.

Although these improvements could help in theory, the next step for this application is to use more modern methods like convolutional neural networks (CNNs) to similarly generate low-dimensional mathematical representations of an input face for a more robust implementation.

Conclusion


The Commit Hackathon Onboarding Project is indicative of the culture at Commit, which promotes learning and collaboration between Engineering Partners in the internal community. Not only do you get a chance to build and learn about something cool, but you can continue to learn by joining future HOP demos. Commit’s Slack community also provides a way to share ideas and ask about a problem that you may be struggling with. One thing’s for sure: Commit places Engineers’ happiness above all else, and they take great care in matching you with startups that are working on problems that align with your interests.

Alejandro Esquivel is a Senior Engineer with front-end and back-end expertise. He began developing websites in elementary school to fund the purchase of his first laptop, and ever since then he’s been hooked on finding business ideas that have a positive impact on society. Currently he’s interested in learning about data science.

We hire Software Engineers to build their careers with promising early-stage startups.

###

Interested in joining our waitlist? Sign up now.
We hire Software Engineers to build their careers with promising early-stage startups. Apply today!