Immersive Scholar
Increased awareness of digital scholars' data visualization art by building web apps for large-scale video walls.
Overview
Immersive Scholar is a project funded by the Andrew W. Mellon Foundation to promote the creation and sharing of digital scholarship in large-scale visualization environments. One key component of the project is an open-source, responsive web app for sharing these visualizations. The app is designed to work on both large-scale, immersive displays as well as personal-sized interfaces. It also contains documentation for optimal usage so other institutions may place their own content in the app to create their own large-scale data visualization displays.
This grant is led by NCSU Libraries at the North Carolina State University (NCSU). As part of the grant, NCSU Libraries hosts and collaborates with resident scholars to create data-driven visualizations. Because each resident's content is unique, each resident will actually have a different app suited to their needs. However, these apps can be easily modified by others that want to create their own versions of the displays with their own data.
NCSU Libraries has produced content with the following scholars:
- Tess-celestial with Radmila Sazdanović
- Coded Glass with Liss LaFleur
- Community Gardens with Lucas Swick
- Surface Tension with Caitlin Foley and Misha Rabinovich
- Untitled with Emil Polyak
I helped design and develop the Tess-celestial and Coded Glass apps, created the logo for Community Gardens, and contributed to the documentation for the overarching Immersive Scholar project.
Process
Tess-celestial
Objectives
Build a responsive web app for sharing data visualizations created by scholars. The web app must be scalable to large-scale displays - such as the video walls at NCSU's Hunt Library - and personal-sized displays (i.e. desktops, laptops, tablets, and mobile phones) alike. It should also be customizable so other institutions can use it to create their own data visualization displays.
For Radmila's visualization, allow users to view her array of tessellations and the story behind each of them.
Ideation
Our first resident was Radmila Sazdanović, an assistant professor of Mathematics at NCSU and one of the creators of Tess, the software she used to draw tessellations. She wanted to showcase tessellations of mathematical concepts we encounter in our daily lives. During our initial team meeting, we decided that the display would have two modes: the overall view and the modal view. The overall view would contain a matrix of all the tessellations to create a colorful, visually interesting display to attract passerby. There would also be a sidebar with a brief description about the project. When one of the art pieces is selected, a modal about it appears on the screen. It would display a larger version of the piece for a more close-up look, the story behind it, and some ancillary images or art pieces related to it. The display would continuously cycle through all the matrix items and their stories.
Now we had about five weeks to build the app from scratch. I started by wireframing different matrix layouts and animating storyboards of the display flow - trying to figure out how the exhibit could convey the idea of "immersion." This led to the "deep dive" animated transition between the overall view and the modal view. When the viewer sees the whole matrix, all they see is the "surface." Then the display "dives" into (zooms into) an art piece to reveal the story behind it. After giving the viewer some time to appreciate the tessellation, the display "resurfaces" from (zooms out of) it back to the overall matrix view.
Development
Next, it was time to code the app. I divided the display into four main components: the project description, the matrix, the modal, and the footer. Those parts were further broken down into groups of text and images. Once the HTML structure was established, I used the CSS Grid Layout to align the aforementioned components. I also used flexbox within those sections and media queries to scale elements appropriately when the display size changed. Lastly, I refactored the stylesheet with CSS variables so it would be easier to read and manage. My teammate, Walt Gurley, wrote the JavaScript that loaded the images and stories as well as animated the "diving" and "resurfacing." The two of us also tested the display on Hunt Library's video walls frequently with the rest of the team so we knew what needed fixing. Some of the challenges we encountered included overflow, lag, and dizziness, meaning we had to keep tweaking sizes and speeds.
After we debuted Radmila's content in the library and new residencies began, Walt and I continued to update the codebase for Tess-celestial. I added media queries for a wider range of large-scale aspect-ratios as well as for personal-sized displays. Unlike our large-scale displays - which had timed animations because the video walls were non-interactive - personal-sized displays needed some extra interactivity to improve the user experience. Thus, I added JavaScript to allow the user to navigate between modals without zooming back out and also view the ancillary images in greater detail.
Results
The Tess-celestial app is designed for two types of displays: non-interactive large-scale displays (e.g. video walls) and interactive personal-sized displays (e.g. desktops, laptops, tablets, and mobile phones). Here, we define "large-scale" as any display with a viewport height (vh) of at least 2100px. Therefore, "personal-sized" means the vh is less than 2100px. This breakpoint is used to determine whether there are automatic transitions or not.
The large-scale display starts with a view of the entire matrix of tessellations, which are randomly ordered each time the app is refreshed. After about 30 seconds, the display chooses a piece to dive into, and its modal fades into view. The modal displays a larger version of the selected piece, 0-4 ancillary pieces or images related to it, and the story behind its creation. The viewer has a minute to digest the info, after which the modal fades out and the display zooms back out to full matrix view. It visits each tessellation once before revisiting any.
The personal-sized display also starts with the full matrix view. However, unlike on the video walls, this display is scrollable because the device is interactive. The user can also choose which pieces they want to view by clicking/tapping on them, and there are no automatic transitions. Within each modal, they can click/tap on the ancillary images to view larger versions of them. They can also click/tap the arrows on the sides to view the modals of the previous and next pieces in the matrix. This makes it easier to view many pieces without having to go back to matrix view each time. Lastly, the user can exit the modal when they're done by clicking/tapping the X button in the corner or anywhere outside of the popup and its other buttons. Alternatively, the user can use the left and right arrow keys to move between modals and the Esc key to exit modals.
Check out the Tess-celestial GitHub repo and web display here.
Coded Glass
Objectives
Similar to Radmila's visualization, users should be able to view a collection of Liss' data-inspired art and the story behind each piece in a responsive web app. However, the visualization also needs to create spaces for reflection.
Ideation
Liss LaFleur, artist and assistant professor of New Media Art at the University of North Texas, was our next resident. She created a series of digital stained glass windows inspired by data from the #METOO movement. Her goal was to create meditative spaces around the video walls by telling the story behind a window series and slowly panning and zooming through it. The viewer would be able to appreciate the art, learn about #METOO statistics, and reflect on the status of #METOO.
Development
We adapted the Tess-celestial app to Liss' needs. The circular matrix items for the index page became rectangular, and we removed the modal view. Instead, we linked to slideshows that Liss created using NCSU Libraries' ImageViewer software. For the large-scale displays, we also added timed transitions between the index page and the slideshows. Eventually, to keep all of the content local, we implemented our own custom ImageViewer via JavaScript and moved all of the slideshow images to the web app.
Results
Like Tess-celestial, the Coded Glass app is also divided into large-scale displays and personal-sized displays via the 2100px vh breakpoint.
The large-scale display starts by displaying thumbnails of the windows in a matrix, which are randomly ordered each time. Each thumbnail represents a different series of windows. The user has a minute to read the project description and appreciate the display before it fades into the slideshow of the first series. Each slideshow begins by showing the title, the story behind the series, and then the windows themselves. Each window has the Ken Burns effect applied to it, making the viewer feel calm as they appreciate the glass art and reflect on the story. After about 45 seconds, the description (story) overlay fades away, allowing the viewer to better appreciate the details of the glass. Once one slideshow is finished playing, the display moves on to the slideshow series of the next image in the matrix. It continues playing all of the series before returning to the matrix page.
The personal-sized display also begins with the matrix, but the user can choose which window series to view. Hovering over a thumbnail shows the series' title. Clicking/tapping a thumbnail will play its slideshow. They can exit a slideshow by clicking anywhere on the slideshow while it's playing. Currently, the slideshow autoplays through each image in the series when clicked on. However, we plan to change this for personal-sized devices so the user can pan/zoom through the images themselves in the order they desire.
Check out the Coded Glass GitHub repo and web display here.
Community Gardens
Objectives
Design a logo for Lucas' Community Gardens installation.
(Lucas is an engineer, so he can develop his own display).
Ideation
We hosted Lucas Swick, a generative artist, as our third resident. Lucas engineered an installation about housing and food insecurity at NCSU using threejs. In it, each plant represented a student, and its physical attributes indicated the stressors it endured. He wanted to bring attention to the hidden struggles of students on campus, challenge the stigma of homelessness, and urge individuals to help those in need.
This time, I didn't code the display, but I did help detail the installation flow and website architecture via storyboards and wireframes, produce visual assets (i.e. plant patterns and mockups), and develop the brand (primarily the logo). When creating the logo, I started with the initials of "Community Gardens." Then I explored visual concepts that conveyed ideas relating to "community," "garden," "data," "generative," and "resilience" while linking the two letters together. As for the typeface, I chose Alegreya because it was "organic" yet "modern" - words that can also be used to describe a digital garden. It was reminiscent of a storybook and inviting, but it was also contemporary.
Results
As in the previous two projects, Community Gardens' large-scale versions are automated, while its personal-sized versions are user-controlled. Because the video walls have more processing power, they contain 3D titles, sidebars, and more plants than in the personal-sized displays. Both versions display a data-driven garden that can be panned through. The physical composition of each plant symbolizes the story of a student and their stressors. To view gardens driven by specific variables, read about the process behind this installation, and learn how to help, viewers can also visit the website on their personal devices.
The final version of the logo consists of the letters "C" and "G" connected with a loop to resemble a vine. Its upwards curl is also reminiscent of a flexed bicep, radiating resilience and hope. A leaf sprouts within the negative space in the loop.
Check out the Community Gardens GitHub repo and web display here.