Generative AI tools that convert image to text are gaining more attention in the recent years. Their highly-anticipated use cases include helping people with sight impairment and captioning images for different contexts such as product description.
I have started to wonder how generative AI views paintings by human artists and In this blog post, I intend to explore the world of art through the eyes of Astica vision AI, a tool that receives images and generates a detailed description of the image in natural language.
Expectations
Before engaging with Astica Vision, here are my expectations. First and foremost, I expect to receive an accurate description, in the sense that no object is misidentified. Additionally, I hope that the tool can provide a description beyond mechanically introducing each element. More specifically, a valuable description takes the painting as a whole into account, rather than viewing it as a combination of separate components. I am also excited to see how certain AI capabilities such as tagging are utilized to generate an output.
Paintings
For this experiment I have selected two paintings with assumably different levels of difficulty for AI interpretation. The first painting, Nighthawks is a 1942 painting by Edward Hopper that depicts “the loneliness of a large city” (Art Institute Chicago, n.d.). The second paining is the Starry Night by Vincent van Gogh (1889).
Experience
Astica Vision exceeds my initial expectations. It generates two major outputs for each painting. One is a GPT-S description that holistically explains the painting while trying to convey its sense in addition to the actual elements. Here is the description of Nighthawks. I have underlined the parts I find the most interesting.
“This is an image of a painting that depicts people sitting at a bar. The scene is mostly dominated by dark hues of black and brown. In the painting, there are a few individuals who stand out. One of them appears to be a man dressed in a suit, sitting towards the middle of the painting with his back to us. Towards the right, there is another man who seems to be in a uniform. Both of these individuals are slightly blurred, adding a sense of motion or mystery to the scene. There is also a counter visible in the painting, which is somewhat blurry, further emphasizing the impressionistic style of the painting. To the left, there is another person, though their features are less defined. There is also some text visible in the painting, specifically the words “PHILLIES Only 54”. Overall, this painting provides an intriguing glimpse into a social setting, possibly a late-night gathering at a local bar.”
The second output, which is called the vision AI output, categorizes, captions and tags the painting in a more concise way, probably more suitable for practical use cases. Take a look at the Vision AI output for Starry Night.
A few points stand out in my interaction with Astica Vision. First, the tone of the description is professional but it maintains a level of uncertainty. This is evident in the abundant use of the words “suggesting” or “appears to be” which can be an advantage since it constantly reminds the user of the uncertain nature of AI. Additionally, the tool provides a confidence score for each of the identified objects, a useful piece of information for the users. Moreover, the tool provides a categorization of safe versus harmful content which can be further applied in use cases targeting children.
Final thoughts
Astica Vision is a powerful tool that can describe and caption images in natural language. While challenged with abstract works of art, it still maintains a high level of performance and generates outputs that exceed expectations.
However, it is important to note that the tool still fails to identify all the present objects separately or the layered meaning of complicated paintings. A task that will apparently remain in the realm of human capabilities for the years to come while AI takes care of more practical aspects such as captioning and tagging.
References
Art Institute Chicago. (n.d.). Nighthawks.
https://www.artic.edu/artworks/111628/nighthawks
Edward Hopper. (1942). Nighthawks. [oil on canvas]. Art Institute of Chicago. Chicago, Illinois/United States
Vincent van Gogh. (1889). The Starry Night. [oil paint]. The Museum of Modern Art. New York City, New York/United states.
