Building User Experience (UX) Capacity to Support Digital Transformation: A Case Study of Cooper Hewitt, Smithsonian Design Museum
Craig MacDonald Pratt InstituteElena Villaespesa National Gallery of Art
Danielle Kingberg
Kyle Kisicki
Rachel Jackson
Abstract
To take full advantage of the potential offered by the digital space, museums and cultural institutions must be able to consistently apply User Experience (UX) methods to create enjoyable and understandable digital interfaces. Unfortunately, many of these organizations lack internal UX expertise, which means they need to partner with costly outside vendors to provide digital expertise, rely on internal staff and struggle through a process of trial and error, or do nothing and fall further behind in their digital offering. The COVID-19 crisis has intensified the digital transformation of museums. Although many museums are welcoming people back into their galleries, visitors’ demand for enjoyable digital experiences will persist. Absent a focused effort to build and sustain their internal capacity to apply effective UX practices, museums risk alienating their visitors with poorly designed interfaces that reduce engagement and decrease traffic in both virtual and physical spaces. Our solution to this problem is to engage in User Experience Capacity-Building (UXCB), a process through which organizations can gradually create a more UX-friendly culture and build a sustainable and effective internal UX practice. UXCB is designed as a highly contextual process; it focuses on identifying and building on each organization’s existing strengths to then address its specific challenges. Conceptually, UXCB includes three stages. The first phase includes securing buy-in and support from relevant project stakeholders, performing a needs assessment to identify the organization’s strengths and weaknesses, and setting achievable and realistic goals. The insights from this stage are then used to inform the second phase, which is about implementing UXCB strategies. This step includes figuring out what UXCB activities will be used, what content they will focus on, what resources will be needed, and when and how they will be implemented. Finally, the third phase is evaluating the outcomes of the UXCB initiative, which can be observed at both the individual and organizational levels. In this paper, we present a case study of implementing a UXCB initiative for Cooper Hewitt, Smithsonian Design Museum. We provide a detailed description of our UXCB process: assessing the museum’s initial UX capacity to determine its needs, developing a UXCB ‘game plan’ based on the museum’s specific needs and context, implementing the various UXCB ‘plays’ outlined in the game plan, and evaluating the impact of the UXCB initiative. We then reflect on the value of the UXCB model and its applicability to museums and share some of our major lessons learned from the experience. We conclude with suggestions for implementing the model in other contexts and directions for future work. The discussion points out practical implications for museums that want to increase and improve their internal UX practices.
Keywords:
1. Introduction
1. Introduction
Over the past decade, museums have greatly increased their digital offerings in multiple functions including marketing, communications, education, publishing, and curatorial. Parry’s (2013) vision of the post-digital museum anticipates the moment when digital is so integrated into museum practices, policies, and strategies that digital transformation becomes naturalized within the museum and permeates the museum’s mission and strategy. While there are some advancements in the museum sector to reach this state (Pryor, 2016; Stack, 2013), recent data questions the readiness of the digital capacity in museums (Knight Foundation & HG & CO, 2020) and the COVID-19 crisis has only intensified these challenges (Noehrer et al., 2021). Though the pandemic has not yet faded from daily life and its long-term effects are far from clear, what seems certain is that visitors’ demand for well-designed digital interfaces that supplement or replace in-person museum experiences is here to stay.
Museums that have already improved their digital capabilities may be best positioned to meet this challenge, but designing beautiful, engaging, and memorable digital experiences requires more than just technical skill: it also requires expertise in the User Experience (UX) design process (Hartson & Pyla, 2019; Sharp et al., 2019). While several museums have successfully adopted UX methods and tools (Kabassi, 2017), the vast majority of museums lack the internal expertise to do so, forcing them to make a difficult choice for every digital project: do they (a) partner with costly outside vendors to provide design expertise, (b) rely on untrained internal staff and labor through a process of trial and error, or (c) do nothing at all and fall further behind in their digital transformation?
Our solution to this problem is to engage in User Experience Capacity-Building (UXCB), a process through which organizations can gradually create a more UX-friendly culture and build a sustainable and effective internal UX practice (MacDonald, 2019). UXCB is a highly contextual process that focuses on helping internal staff identify their organization’s existing strengths to then address their specific challenges. Conceptually, UXCB includes three stages, as depicted in Figure 1 below. The first phase is focused on establishing the right conditions for UXCB to succeed. The insights from this stage are then used to inform the second phase, which is about implementing UXCB strategies. Finally, the third phase is evaluating the outcomes of the UXCB initiative, which can be observed at both the individual and organizational levels. In turn, this improved capacity creates new conditions for the next UXCB initiative, thus creating a continuous cycle of organizational improvement.
UXCB has been shown to be effective in creating a more user-centered culture in large technology firms (Stone et al., 2016), but museums pose a unique set of challenges due to their budgetary constraints, their limited number of dedicated UX staff, and varied levels of in-house technical and design expertise. As a result, there is a clear need to develop, evaluate, and promote effective ways for museums to overcome these challenges and build their internal UX capacity.
In this paper, we present a case study of implementing a UXCB initiative for Cooper Hewitt, Smithsonian Design Museum. We start with an overview of Cooper Hewitt to provide context for the case study. Next, we provide a description of our UXCB process, explaining each of the three stages: conditions, strategies, and outcomes. We conclude by reflecting on the value of the UXCB model and its applicability to museums. share some of our major lessons learned from the experience, and outline our plans for future work in this area.
2. About Cooper Hewitt, Smithsonian Design Museum
Cooper Hewitt, Smithsonian Design Museum, was founded in 1897. Located in the landmark Carnegie Mansion in New York City, the museum offers four floors of galleries containing more than 215,000 objects covering over 3000 years of design history, including drawings, prints and graphic design, product design and decorative arts, and textiles and wall coverings. As of FY2020, the museum had an annual operating budget of approximately $15 million and roughly 80 full-time employees. In terms of visitors, the museum’s website receives 1.3 to 1.4 million visitors per year; prior to COVID-19, the museum attracted between 250,000 and 400,000 in-person visitors each year (Smithsonian, n.d.).
The museum has invested in its digital offerings over the past few years. In 2011, the Digital & Emerging Media department was established, and upon reopening in 2014, they introduced a new website providing access to 215,000 collection items, along with an innovative on-site experience centered around “The Pen”. Upon entering the museum, visitors were given a digital pen that they could use to either collect objects by tapping on their labels or to design their own wallpaper, furniture, and objects by following the prompts on the interactive tables. After their visit, people could access these objects and creations at home using a unique URL provided during their visit (Chan & Cope, 2015). About a third of the visitors engaged with this experience and checked their content after their visit (Chan, 2019). The museum’s efforts also include making their collections accessible, first with a public API and then with the launch of the Smithsonian Open Access initiative.
The user experience remained a priority for the museum and that was reflected in the appointment of their first Chief Experience Officer in 2018, a role responsible for both the on-site and digital experiences (Cooper Hewitt, Smithsonian Design Museum, 2018). Under this role the museum created the Interaction Lab (The Interaction Lab, 2019) to imagine and develop future museum experiences. The museum digital efforts also included opening their collections, first with a public API and then with the launch of the Smithsonian Open Access initiative (Smithsonian Open Access, n.d.).
3. Building UX Capacity at Cooper Hewitt
In this section we will detail our approach to helping Cooper Hewitt build their internal UX capacity over a 9-month period, from May 2021 to January 2022, following the three phases of the UXCB process: conditions, strategies, and outcomes.
Phase 1: Conditions
Assessing the museum’s UX capacity
As the first step, we set out to gain a deep understanding of the museum’s current UX capacity. The assessment was conducted during a 90-minute remote interview (via Zoom) with the Chief Experience Officer, the Digital Product Manager, and the Director of the museum’s Interaction Lab on May 24, 2021. After an overview of UXCB, we asked the Cooper Hewitt team to respond to a series of questions based on the UX Capacity Assessment Framework (MacDonald et al., 2021). This framework includes 21 components of UX capacity split into six dimensions: people, resources, practices and processes, organizational literacy, organizational decision-making, and benefits. After the interview, we summarized the team’s responses to each question on Miro, an online brainstorming tool, and coded their responses as either a strength, weakness, or neutral. We summarized our results by identifying the museum’s assets (strengths) as well as the most prominent obstacles (opportunities) for each dimension of UX capacity.
Developing a UXCB Game Plan
Once the assessment was complete, our next step was working with our Cooper Hewitt stakeholders to collaboratively develop a UXCB ‘Game Plan’ that outlined the goals, activities, and a timeline for implementing UXCB at the museum. Using collaborative brainstorming and discussion, we identified 15 potential UXCB ‘plays’ (i.e., activities), split into the seven main types of activities identified by MacDonald (2019): training workshops, technical assistance (i.e., coaching), hiring, team building and structuring, events, broadcasting, and guides, tools, and frameworks.
Two weeks after our preliminary interview, on June 7, 2021, we held a 90-minute virtual workshop with our three museum stakeholders to share the results of our assessment and our list of proposed UXCB plays. After a brief presentation of our findings, we used a shared Miro board to facilitate a collaborative brainstorming exercise to identify which activities our museum stakeholders were most interested in implementing. From these results, we developed a draft UXCB Game Plan that included a summary of our assessment results, a list of proposed principles to guide the museum’s UXCB work, and our recommended UXCB ‘plays’ along with a proposed implementation timeline. We presented our draft game plan to our museum stakeholders in another 90-minute workshop on July 6, 2021. Based on feedback gathered during this workshop, we crafted a final game plan that included three ‘pillars’ to guide UXCB work at Cooper Hewitt:
- Learn: Build the digital team’s confidence by leveling up their ability to apply good UX practices on digital projects
- Assist: Provide targeted feedback and suggestions to help overcome challenges and stay on the right track while going through a UX project; and
- Showcase: Help craft case studies that increase stakeholders’ appreciation for UX.
For the first pillar (Learn), we proposed a play called “UX Fundamentals” focused on teaching basic UX concepts and another called “User Testing Bootcamp” focused on teaching staff how to conduct usability tests. For the second pillar (Assist), we proposed a play called “UX Coaching” that would include a series of regular, planned meetings with the digital project manager to provide knowledge and resources as needed. Finally, for the third pillar (Showcase), we proposed a “UX Storytime” play, a training workshop focused on teaching the project team about UX storytelling techniques and how to craft a compelling case study of the project.
Phase 2: Strategies
Importantly, because capacity-building efforts are context-dependent, they often require adjustments and changes in order to be successful (Labin et al., 2012). Therefore, we were not surprised that the constantly shifting environment both within and outside Cooper Hewitt forced some modifications to our plan, with the most significant change being that we decided not to run the “UX Storytime” play due to time constraints and scheduling challenges. As a result, we ended up running three UXCB plays: UX Coaching, UX Fundamentals, and User Testing Bootcamp, each of which will be described below.
UX Coaching
The “UX Coaching” play involves holding regular meetings (weekly or bi-weekly) between a UX expert, who may be internal or external to the organization, and a museum stakeholder (or group of stakeholders). Depending on the needs of the project, additional stakeholders may be invited to one or more sessions. The purpose of the sessions is to provide UX-related advice, support, and mentoring for new or ongoing projects. While sessions can have a set structure or agenda, they may also be un- or semi-structured to allow for spontaneity and flexibility around the stakeholder’s priorities and challenges.
In our case, the UX Coaching play with Cooper Hewitt consisted of 15 one-hour unstructured coaching sessions with the Digital Product Manager (DPM). Sessions were held roughly once per week from August 31, 2021, to January 25, 2022 (with some breaks for holidays). All sessions included the DPM and the entire research team, though at one session the DPM also invited the head of the museum’s visitor experience department to discuss how their staff could participate in the UXCB project.
UX Fundamentals Workshop
The “UX Fundamentals Workshop” play is a training workshop to introduce museum staff to foundational UX principles and concepts. This workshop defines UX as a holistic paradigm and advocates for its relevance and value. In addition to short lectures, it should include interactive elements (i.e., polling and breakout room activities) to generate interest and discussion among participants. The workshop can be held in-person or virtually and should take 60-90 minutes to ensure sufficient time to cover the material and complete the interactive activities.
We implemented the “UX Fundamentals Workshop” play as a 90-minute virtual workshop (hosted on Zoom) on September 28, 2021. We designed the workshop for a general audience of about 25 museum staff members with varying levels of UX familiarity. Our purpose was not only to introduce staff to UX but also to establish UX as a strategic priority in building a visitor-first culture at the museum in a way that was neither daunting nor impractical.
The session began with an interactive “mythbusting” activity using Zoom’s polling feature in which we gauged participants’ preconceptions about what UX meant in the context of their work. Following the myth busting activity, we explained the general stages of a UX design process and included two activities to give them hands-on experience. The first activity gave participants the opportunity to develop empathy maps for four distinct predetermined visitor personas from their museum using the collaborative whiteboard tool Miro. The next activity built upon the insights from the previous one: brainstorming users’ goals in visiting the museum, discussing their pain points, and establishing how these goals present opportunities for Cooper Hewitt to design for their needs. After a short discussion about these activities, we turned to establishing the value of implementing UX methods by highlighting case studies and opening the session for audience questions.
User Testing Bootcamp
The “User Testing Bootcamp” play is an interactive workshop to introduce and train museum staff on how to plan and implement a moderated usability study. This workshop builds on the attendees knowledge of UX fundamentals by walking them through the process of planning a simple user test of one of the museum’s digital products (i.e., the website). This workshop is meant to be highly interactive, featuring breakout room activities followed by Q&A with workshop facilitators. The workshop can be held in-person or virtually and should take at least 90 minutes, though two full hours is recommended.
We implemented the “User Testing Bootcamp” play as a 90-minute virtual workshop (hosted on Zoom) on January 12, 2022. It was also designed for a general audience of about 25 staff members, some of whom also attended the previous UX Fundamentals Workshop. The session began with a quick overview of UX fundamentals to ensure that all attendees understood what UX is before we moved forward. Next, we introduced user testing within the context of the UX process and shared the steps involved to plan and conduct a user test. The session included a mix of lecture content and interactive activities tied to different steps of the user testing process. For example, one activity asked workshop attendees to brainstorm different ways to recruit potential users for different audiences (e.g., families, site-seers, design experts). We also split attendees into smaller groups to design and pilot test a simple usability study of the museum website, with one of the workshop facilitators acting as the ‘user’ and another acting as the moderator. We concluded the workshop with a brief discussion of how to be a good moderator and some approaches for efficient data collection and analysis.
In our post-workshop debrief, we identified two areas for improvement for this play. First, we budgeted 90 minutes for the session but ended up going over the scheduled time. Therefore, we recommend planning for at least two hours. Second, we identified the pilot testing activity as the most impactful in terms of creating buy-in around the value of user testing because it was the first time most workshop attendees had ever observed someone using their website in real-time. As a result, in future iterations of this play we plan to move the pilot testing activity earlier in the session and also devote more time to the activity.
Phase 3: Outcomes
Next, we set out to gather data about how our work impacted the museum’s UX capacity. In this section, we describe our methodology and then summarize findings related to our implementation of each play and the impact of the entire UXCB initiative, including outcomes at the individual and organizational levels.
Methods
We used two different methods to evaluate the UXCB plays. To evaluate the workshops, we designed a survey in SurveyMonkey and distributed it via email on February February 8, 2022, to all museum staff who attended at least one of the workshops. The survey asked respondents to indicate which workshop(s) they attended and included questions about the workshop’s content, presentation, and overall quality. We also included a series of questions asking respondents to compare their UX interest, knowledge, understanding, and confidence before and after attending the workshop(s). We received 12 responses to the workshop survey, which included 6 people who attended both workshops, 3 who only attended the UX Fundamentals workshop, and 3 who only attended the User Testing workshop. Therefore, we received 9 responses about each individual workshop, which accounts for roughly one-third of each workshop’s attendance. Respondents represented a range of museum departments, including content and marketing, visitor experience, advancement, library, education, and daily operations.
To evaluate the UX Coaching play, we conducted an exit interview with the DPM about the effectiveness of the coaching sessions, their attitude towards UX, and whether they had acquired new UX knowledge, skills, or behaviors as a result of participating in the sessions. Also, as our key stakeholder throughout the project, we also asked the DPM to assess the impact of the entire initiative on Cooper Hewitt’s UX capacity.
Implementation Results
Regarding the UX Fundamentals Workshop, respondents provided an average rating of 4 on a 5-point scale where 1 was “poor” and 5 was “excellent.” One respondent noted it was a “good introduction to the concept [of UX]” while another said it was “well organized and easy to follow.” For the User Testing Bootcamp, respondents provided an average rating of 4.22 on a 5-point scale where 1 was “poor” and 5 was “excellent.” One respondent said it had “great content, great breakout sessions” while another said it was “extraordinarily eye opening how doable ‘UX’ is.” Overall, respondents felt both workshops were relatively easy to follow, that the content was presented clearly, that the interactive exercises were beneficial, and that the presenters were well prepared. Further, 100% of respondents said they would recommend both workshops to other museums.
Regarding the UX Coaching sessions, the DPM reported finding these sessions useful and informative. At the outset, the goal of these sessions was to help the DPM implement a UX process on one of the museum’s ongoing projects. However, as the museum’s focus shifted away from this project, the sessions became more general and touched on a variety of topics, including identifying other UX champions within the museum, establishing a UX framework for all museum projects, and planning the other UXCB plays involving other museum staff. Overall, the DPM said that although they found these unstructured discussions beneficial, they would have been even more impactful if we had focused on achieving a specific outcome, such as getting more buy-in and support from staff throughout the museum.
Impact
According to the conceptual model (Figure 1), UXCB outcomes can occur at both the individual and organizational levels. When evaluating individual-level outcomes, our focus was on (1) the attendees of the two workshops and (2) the DPM, who was our key stakeholder and participant in all the UX Coaching sessions. For the workshop attendees, we used the retrospective pretest method to evaluate the impact of the workshop on their attitudes towards UX by rating each area before and after attending the workshop(s). Using a paired t-test (p < 0.05), we found that respondents indicated that their interest in UX, knowledge of UX concepts, and confidence in using UX methods all increased after attending one (or both) of the workshops (see Table 1). While their understanding of how UX applies to their museum work did not increase significantly, we suspect this was mainly because these ratings were already high heading into the workshops and thus there was less room for improvement.
| n | Before | After | p-value | |
| My interest in UX: | 10 | 3.00 | 3.70 | 0.0248 * |
| My knowledge of UX concepts: | 10 | 2.70 | 3.60 | 0.0414 * |
| My understanding of how UX applies to my work in the museum: | 10 | 3.10 | 4.00 | 0.0543 |
| My confidence in using UX methods (empathy mapping, usability testing, etc.) | 9 | 2.56 | 3.56 | 0.0278 * |
In addition, 60% of respondents (6 of 10) said they were already doing or planning to do things differently as a direct result of participating in the workshop(s). When asked to elaborate, all of the respondents noted they were starting to adopt a more audience-focused mindset and approach in their work. As one respondent explained: “As we think about refreshing the Membership page on our website, we are thinking more about it from an audience perspective and considering feedback opportunities.” Other responses included “embed[ding] UX-focused approaches into our content and tool development” and “think[ing] holistically about the visitor/user journey to find appropriate opportunities to communicate our mission and ask for support.”
In the exit interview, the DPM mentioned gaining both more UX knowledge and more confidence in applying UX methods over the course of the initiative. One key moment happened during the 10th coaching session when the DPM opened up a blank Miro board and drafted a UX framework for the museum. Our team watched, in real-time, as the DPM added key principles and guidelines for each step of the UX process, which they had synthesized from earlier coaching sessions and from the UX Fundamentals Workshop. For example, they listed several principles for the “Persona Building” step, including “don’t start from scratch,” “personas are informed by data” and “earlier is better.”
Assessing the impact of UXCB at an organizational level is inherently challenging because any type of organizational change is difficult, time-consuming, and can be a years-long process (Cameron & Green, 2009). But as a preliminary effort to understand the potential organizational impact of our UXCB work, we asked the DPM to consider the 26 obstacles we identified in our initial assessment and categorize them as follows: improved, no change, and worsened. Despite our somewhat limited engagement with the museum, the DPM identified improvements in 17 of the 26 areas (65%), with the remaining 9 areas categorized as “no change.” These results are summarized in Table 2.
| Improved | No Change | Worsened |
| 1. UX roles are not explicitly defined on projects 2. Outside vendors typically work independently of the DEM department“ 3. Everybody is doing UX, so nobody is” 4. No staff members with formal UX training 5. There are content and information “silos” throughout the museum; little integration between different departments 6. No design goals/principles, personas, journey maps, or user satisfaction metrics 7. Lack of overhead management for UX work 8. No dedicated usability testing process 9. No overall audience strategy 10. UX work is not typically visible to the rest of the museum 11. UX work is not purposefully planned or organized (“everybody’s winging it”) 12. UX work is not coordinated or integrated across different departments 13. Other museum leaders do not fully grasp the scope and value of UX 14. The website has no clear owner; marketing has heavy influence on content 15. Accountability for UX decisions is not always clear 16. Design decisions are sometimes expert-led rather than based on sound UX knowledge or insights 17. Not much re-use of methods or materials from project to project | 1. No staff members with an explicit UX role 2. No formal funding stream or model for UX work 3. Finances overall are “unstable and unsustainable” 4. Style guide is limited to brand guidelines for the web 5. Some alignment with marketing processes, but process is print-focused and not collaborative 6. Audience assumptions are sometimes framed as audience knowledge 7. Some user satisfaction data is anecdotal and based on project owner expertise 8. Website changes are distributed and usually reflect desires of individual project owners rather than overarching strategy 9. UX work is constrained by legacy systems | |
4. Conclusions and Lessons Learned
Based on the results presented above, we conclude that we were able to successfully help Cooper Hewitt increase their internal UX capacity. Although there were many variables specific to Cooper Hewitt that influenced our work, this is actually a feature, not a bug, of UXCB: it is explicitly designed to both identify and build on each organization’s unique strengths and constraints while also efficiently utilizing whatever resources may be available. Thus, we are confident that our three-part UXCB model can be successfully applied in other museum contexts and encourage the museum community to explore ways to both adapt our UXCB plays for their own use and also develop new plays of their own to address different types of challenges unique to their organization. To aid this process, we offer the following lessons and suggestions based on our experience so far.
First, start by identifying one or more museum staff members to become UXCB champions. In our case, the DPM was not just our primary contact person at the museum but also became a vocal leader and advocate for our UXCB work throughout the museum, which was integral to the success of our project. Second, involve multiple stakeholders in UXCB as much as possible rather than targeting a single person or a single department. For us, although the DPM was the only staff member who participated in the coaching sessions, we were also able to reach several other museum staff members through the workshops, which ensured the results of our UXCB work spread to multiple departments. Third, although UXCB initiatives should involve different types of plays, they should still be planned together and connected as much as possible such that each play supplements or complements the others. For example, we not only connected the UX Fundamentals Workshop with the User Testing Bootcamp, we also used the Coaching sessions to plan those other plays and refer back to concepts and techniques mentioned in them, which helped the DPM make connections and gradually increase their UX knowledge over time. Fourth, flexibility and adaptability are essential to the success of any UXCB initiative. The Game Plan is a valuable tool for setting expectations and providing an initial timeline for UXCB activities, but it should be treated more as a guide than an ironclad plan. Museums are constantly facing a variety of challenges, from new technologies to staff turnover to leadership upheaval and changes in strategic direction. A successful UXCB initiative must be adaptable to these changing circumstances, and UXCB leaders must be open to changing plans or updating timelines if needed.
The central premise of our work is that all museums and other cultural organizations can benefit from building their internal UX capacity. While our initial results are promising and we believe that our UXCB model is widely applicable, we acknowledge the need for additional research in this area to better support the museum community in their UXCB efforts. Moving forward, we plan to apply our UXCB model in different types and sizes of museums in order to design and evaluate new and alternative UXCB plays to address a wider range of museum-specific challenges and obstacles to building a robust internal UX practice. Eventually, we hope to create an interactive playbook of proven UX capacity-building strategies specifically designed to help museums and cultural institutions deepen their knowledge of UX concepts, effectively apply UX methods, and establish a stronger UX culture.
References
Acknowledgments
The authors would like to thank the following staff members at Cooper Hewitt for their participation and contributions to this research: Carolyn Royston, Adam Quinn, Rachel Ginsberg, Pamela Horn, and also employees of the museum who participated in the workshops.
We would also like to thank Graduate Assistants Aimee Shi, Kate Nadel, Nora Gordon and Hriti Prashant Shah for their research work during the project.
This research was funded by Pratt Institute, School of Information Faculty Innovation Fund.
0 Craig MacDonald Pratt Institute
0 Elena Villaespesa National Gallery of Art
0 Danielle Kingberg
0 Kyle Kisicki
0 Rachel Jackson
11081 [pods name="Paper" template="user_block" Where="_mw_paper_proposal_id=11081"][/pods]
