Tools for measuring network impact

Useful evaluation can deepen connection to community, establish sustainable change processes, and help to identify opportunities to better leverage human and financial resources. At Pillar, we take care to embed evaluation into the majority of our programs and initiatives. We have adopted a systems change mindset for our evaluation work. A systems mindset recognizes that our organizations are connected to a vast network of human and natural systems, and our evaluation efforts measure our contributions to these systems.

Recognizing our interconnectedness has helped us to make strategic decisions that serve our community. As a network organization, it is critical for us to evaluate our impact, failures, and connections to ensure we are meeting the needs of our members and stakeholders. We use the tools listed below measure our effectiveness and identify areas of improvement. Through sharing our evaluation processes we aim to inspire and enable our community to conduct their own useful evaluation practices and improve their organizational impact.

Measuring impact in our network and membership base 

Impact and failure reports

What began as a traditional annual report reflecting on our programs and services has transitioned to be an impact and failure report highlighting our learning, failure and impact as a network. Take a look at our past impact and failure reports

Membership surveys  

We survey our members bi-annually to gather their feedback on their overall satisfaction, member benefits and what else we can be doing to support them. More recently, we began asking our members questions on the Innovation Works chalkboard, shown below, and on our online community to encourage more ongoing feedback. These questions are intended to be more philosophical and thought provoking for those who answer and those who read them.

Network mapping

Pillar has recently engaged in a partnership with The CutlurePlex Lab at Western University to map our network using data visualizations. Network mapping has helped us to establish a baseline so we can monitor our network evolution to compare network pre- and post- our membership redesign and inform our strategy. We will be continuing to examine our overall network transformations. We are testing this model with a new program CityStudio London to track the strength of the relationships and the increase in the relationships for students, faculty and partners from the outset of the program to evaluate the change and impact in relationships. See an example of network visualization analysis in this overview of annual Pillar events from 2011 to 2018. 

Setting a baseline for understanding board and staff impact 

With our strategic focus to increase the understanding and use of impact measurement, our board and staff completed an impact measurement baseline survey that we could use to monitor over the three years of this as a priority area for our organization.

Measuring the impact of our board 

Strategic plan

Every three years Pillar does a strategic plan that engages our network including members, board, staff and partners to set our direction and meet community needs. This process is a combination of design jams with our members and the community, surveys, staff input sessions, board and staff planning days and reflecting on the past so we can move forward. To see how Pillar used our network approach to build our most recent strategic plan watch the video below. 

Board action plan

As part of our strategic planning process, a unique approach Pillar has adopted is creating an annual board action plan that outlines their actions, timeline and who will be responsible to hold the board accountable to the actions they outline. 

Our executive director provides a report to the board at each board meeting summarizing human resources, financial management, government relations/advocacy, partnerships/collaborations, fund development, communications/public relations, program/project highlights, staff action plan progress, monitoring updates related to staff complaints, member complaints and receiver general submissions, and reflections/learning from the ED.

Executive director evaluation

A key responsibility of a board is the hiring and evaluating of its executive director. The Pillar board conducts a full 360 ED evaluation bi-annually where it reaches out to board, staff and community members. The process also includes a self-evaluation by the ED. Our board works with DecisionWise, a tool that evaluates a leader on leadership competencies that are shared across different industries and shares the comparison to other leaders across North America. The years in between the 360 evaluation, the board completes a performance evaluation that is shared with the ED. Each year the feedback from the evaluation is used to create a performance summary that outlines annual development goals.

Board evaluation

An annual board evaluation is a promising practice that enables boards to reflect on their performance, engagement and impact. Together the board creates a summary  of the key themes that emerged throughout the year and they discuss the next steps and actions to address the opportunities and challenges that emerge from the survey results. 

Prompting questions

At the end of each meeting we have a generative question “Have we integrated an equity lens into our discussions? Into our community discussions?” to increase the understanding and practice related to equity and inclusion for our board.

Measuring our staff impact 

Staff evaluation

Each staff member receives an evaluation that includes a self-evaluation, co-worker evaluation and an evaluation and discussion with their direct supervisor. The results of a staff evaluation should not come as a surprise to the staff member as continuous feedback, recognition and get real conversations happen on a regular basis.

Staff action plan

An annual staff action plan that aligns with the strategic plan and has the objectives, tactics and actions for each team member is the guide for our employees throughout the year. We use the RACI method to identify who is responsible, accountable, consulted and informed for each action. We can also sort the plan according to which team cluster is responsible. This has become a foundational tool that is used to track activities and then report to the board.

Staff impact report

The staff impact report is completed by staff monthly and is used to update the board in the ED Report. In the impact report we have cluster reports, cross-cluster reports and action plan reporting.

Indicators tracking

We have an impact indicators tracking sheet that our team completes monthly with impact stories, inclusion moments, learning and development, media, testimonials, volunteer hours and specific indicators for each team cluster. These are rolled up and shared in the impact report and ED report.

Risk management matrix

As our organization has grown, and taken on ownership of a building and additional risk, we adopted a risk management matrix that is provided to the board quarterly. The matrix flags risks according to program and project areas and identifies the risk with green for low, yellow for medium and red for high risk.

Adaptive cycle

The adaptive cycle is a model that Pillar has modified to use as part of our strategic planning process. It is based on the concept of an ecosystem and how it goes through a natural pattern of change from birth to growth, and from maturation to creative construction. The model can also show how resilient an ecosystem, organization, program or project is. In our most recent strategic planning process, we identified that we had many programs that were in birth and growth and that our focus would be to move those to maturation as we get ready for the next stage of growth.

Decision making tool

Another tool that we developed as a result of our most recent strategic plan was a decision making tool. We have a team that is always eager to embrace opportunity and we recognized that we had a need for a tool to guide us through decision making in a time when one of our strategic priorities is “Be Focused”. In a world, and work culture, which values innovation and growth, the word ‘no’ had become almost taboo. Turning down an opportunity or request can be viewed as a hindrance to progress and cooperation. Yet, counterintuitively, we have come to learn that saying ‘no’ is often necessary and can be one of the best ways to provide support.

Prompting questions

As Pillar started to embrace failure within the organization, the ED added an agenda item at the team meetings for staff to share a mission moment and a failure they had experienced. After we had done that for a few years, we decided to shift that to a prompted question on our Basecamp, our shared project management platform. Each Friday, the team is prompted with a question to share any mission moments or failures from the week. 

Additionally, during our executive directors’ reflective practice and research fellowship, there were weekly prompting questions and the answers were shared with the full staff team to have them be part of the learning and experience. The questions included:

  1. Have you noticed themes emerging in your analysis, interactions, research?
  2. Are you blocked on anything?
  3. Has anything surprised you so far?

Program and services evaluation

In our early days, Pillar would evaluate the results of a program at the end of that program and assume that the goals and objectives set out when we started would remain  the same. Today, we believe in developmental evaluation that recognizes complexity and shifts that will emerge and allows for exploration, development and innovation. We now see that the path and destination for each of our programs and projects evolves over time and we need to monitor and sometimes shift  goals. Additionally, the evaluation model for each program or service must usually vary from one to the next as each initiative will have different goals and criteria. Take a look at the links below as a sample of some of the tools we have used to measure the impact of our diverse programs and services.