At this point in my tech career, I wanted to find ways to extend my interests in web development, user experience, and emerging technologies into a single project. There are many ways to encapsulate an emotion, and I wanted to see how I could use user's emotion to create a unique digital experience.
I used React for the frontend, with face-api.js used to detect the facial expressions. On the backend, I used Node.js with the OpenAI API to generate poems. Both applications are deployed on Heroku.
Firstly, I created the frontend application, getting the webcam element connected and able to take a screenshot. From there, I was able to to connect the Computer Vision library face-api.js. There is a lot of flexibilty and options with that package, and I can see it being useful in the future for more projects. For now, I used it's facial expressions model to determine the user's expression in their given photo.
From there, I started on the backend Node.js server. It recieves the facial expression from the frontend, and uses that to create a prompt to send to OpenAI to generate a poem specific for that user. The parameters are set so that outputs are reasonably unique. Once the poem is generated, it is returned to the frontend where it is displayed.
Feel free to also check out my code for the API.
- Further enhance the styling of the application, especially based on the user's emotion
- Experiment with the prompt sent to OpenAI to get even better responses