Metrics and Agency

Public forum for topics that don't fit into the other categories.
User avatar
SilentDawning
Posts: 7348
Joined: 09 May 2010, 19:55

Metrics and Agency

Post by SilentDawning » 20 Aug 2020, 07:26

I have been subject to a lot of situations in leadership (both in and out of the church) where I am responsible for certain metrics.

Often these metrics are a result of the agency of other people. The old home teaching program is a good example. We were responsible for seeing families, but the success of this metric was based on whether the family would let us see them. Under the new ministering program, stake leaders are likely held accountable for whether or not ward priesthood leaders are holding personal priesthood interviews. But the success of this metric depends on whether the local priesthood leaders decide to hold these interviews. It also depends on whether ministers in the quorum are willing to accept a personal priesthood interview.

Even now, Wards are held accountable for baptisms when the non-member has a strong influence over whether they actually get baptized. Same with "percent of endowed members with current temple recommends". This metric is heavily dependent on whether endowed members have decided to renew their recommends.

Is it fair to hold leaders accountable for metrics that depend on others exercising their agency a certain way? How do you measure results FAIRLY when success of these measures depends so much on the agency of others, and not only the agency of the person/leader being held responsible for those metrics?
"It doesn't have to be about the Church (church) all the time!" -- SD

"The individual has always had to struggle to keep from being overwhelmed by the tribe. No price is too high to pay for the privilege of owning yourself."

A man asked Jesus "do all roads lead to you?" Jesus responds,”most roads don’t lead anywhere, but I will travel any road to find you.” Adapted from The Shack, William Young

User avatar
nibbler
Posts: 4520
Joined: 14 Nov 2013, 07:34
Location: Ten miles west of the exact centre of the universe

Re: Metrics and Agency

Post by nibbler » 20 Aug 2020, 09:46

I want to take one step back.

It's a church, right? Why are we measuring people against metrics and holding them responsible in the first place?
Cure sometimes, treat often, comfort always.
— Hippocrates

Roy
Posts: 6217
Joined: 07 Oct 2010, 14:16
Location: Pacific Northwest

Re: Metrics and Agency

Post by Roy » 20 Aug 2020, 10:07

I am fairly involved with my local toastmasters club. There are similar metrics. How many members a club has. How many members are progressing. How many club officers attend trainings. Etc. Etc.

For a global organization, metrics are an easy way to differentiate what areas are growing and thriving from what areas might need extra help.

Unfortunately, when emphasized too much these metrics become the focus of the leadership. If a CEO is judged by the stock price, (s)he might do things that are unwise long term but give a temporary boost. If the teacher's evaluation is heavily weighted by standardized test scores, (s)he might teach almost exclusively to the test content. I have witnessed arm twisting and outright fabrication in order to achieve metrics. It can be a real problem. Getting over the goal line can take priority over more organic relationships and "pastoral care."
"It is not so much the pain and suffering of life which crushes the individual as it is its meaninglessness and hopelessness." C. A. Elwood

“It is not the function of religion to answer all the questions about God’s moral government of the universe, but to give one courage, through faith, to go on in the face of questions he never finds the answer to in his present status.” TPC: Harold B. Lee 223

"I struggle now with establishing my faith that God may always be there, but may not always need to intervene" Heber13

Jabraso
Posts: 14
Joined: 16 Jun 2020, 17:33
Location: UT

Re: Metrics and Agency

Post by Jabraso » 20 Aug 2020, 12:42

I do think it's important to measure things and set goals for improving things. But it's a problem when the numbers become the goal or the wrong numbers are emphasized. We need to put what is must important at the top and measure the things we can that relate to it. For example if making and keeping temple covenants are the true end goal, then endowed members without a current recommend would be good to measure because it can show a trends that need to be investigated. The issue comes when the focus becomes raising the number by any means necessary in stead of figuring out if there's a bigger reason behind a downward trend in an area.

Numbers for their own sake became a big issue for me on my mission and that was really disenchanting. I had one companion in my second area who was laser focused on numbers. We were in the area that historically had the worst numbers across the board (for years), but once this elder got there the area "miraculously" had the highest numbers of lessons taught for the entire mission. We did work hard, but sacrificed "quality for quantity", and obviously nothing real came of it. Then when that companion moved on (with a pat on the back and glowing praise from mission leaders) things went back to normal when my next companion came in (guess which companion I became good friends with and am still in touch with and guess how I was seen by mission leaders after that).

Arrakeen
Posts: 138
Joined: 25 Aug 2018, 18:49

Re: Metrics and Agency

Post by Arrakeen » 20 Aug 2020, 16:12

“When performance is measured, performance improves. When performance is measured and reported, the rate of improvement accelerates.”
This is the quote in Preach My Gospel that they always beat us over the head with.

Yes, “performance” improves when measured and reported. But what is “performance”? What are we actually measuring? Let’s take baptisms. Are you measuring the number of people coming to Christ? No. Number of people who will be lifelong members of the church? Also no. You are simply measuring how many people get submerged in a font. The problem that comes up, not only in church but also in business and education, is that usually the things that matter most are hard to measure. So we use easy to measure metrics. If we remember that we’re not actually measuring what we care about, we can be careful about how we interpret the numbers and keep focused on what really matters.

There’s also something called a perverse incentive. There’s an example where the British in India wanted to control the cobra population, so they offered people money for bringing in dead cobras. But then people started doing things like breeding cobras to make more money. Sometimes our focus on numbers can do similar things and lead to “baseball baptisms” and such.

As for how to measure, there’s not a really good way to measure that’s not at all based on other people’s agency. Maybe we should have opinion surveys that ask people how satisfied they are with their leaders. Ask if they feel their bishop is kind, respectful, forgiving, etc. Ask if they feel welcome. Solicit feedback, and try to improve on those metrics.

Actually, in college I find that the best measure of a professor is often the reviews on ratemyprofessor. Not test scores, publications, or anything else. We could crowdsource our metrics. Have missionaries have a QR code on their pass-along cards that lets their contacts take a survey about them. Were they pushy? Polite? What is your overall impression? Did they make you more or less interested? Offer a chance to win a gift card to get people to take the survey. Ok, that might be a little too far but I think the general concept of crowdsourced metrics could be interesting. I have taken similar surveys about the quality of service at the grocery checkout line, so some organizations do use this kind of thing.

User avatar
SilentDawning
Posts: 7348
Joined: 09 May 2010, 19:55

Re: Metrics and Agency

Post by SilentDawning » 21 Aug 2020, 07:36

When I posted the opening post, I was thinking more about the way leaders are held accountable. How can it be done with fairness when it is the agency of others that often determines the metric? How do you protect yourself as a leader when people above you simply look at low numbers and assume you are doing a poor job? The latter happened to me a lot in a Ward with low functioning priesthood members. In one Ward, I was sort of raked over the coals when we reported 99 prospective elders and only 1 active. Yet we had visited every one of them in the last 6 months to encourage activity and had made it a big area of focus.

Upon reflection, I think it's a matter of reporting the result metrics AND the process/activity efforts in pursuit of those metrics.

For example, if you are a stake leader/high councilor/stake presidency member responsible for seeing the EQ presidents are holding PPI's, you would report two things. You would report the number of PPI's held by the EQ presidency, AND your own efforts to motivate the EQ presidency. This would mean appending your report with activity such as a) having held a meeting with the EQ president to go over PPI's b) training you gave the EQ presidency in how to approach PPI's c) the plan you asked the EQ Presidency submit showing their schedule for holding PPI's.

If after all that activity the EQ still had low PPI's, the Stake President, or person monitoring you, would know that the low numbers were due to the agency of the EQ, and not lack of leadership from the High Councilor or Stake President's Counselor.
Last edited by SilentDawning on 21 Aug 2020, 07:40, edited 1 time in total.
"It doesn't have to be about the Church (church) all the time!" -- SD

"The individual has always had to struggle to keep from being overwhelmed by the tribe. No price is too high to pay for the privilege of owning yourself."

A man asked Jesus "do all roads lead to you?" Jesus responds,”most roads don’t lead anywhere, but I will travel any road to find you.” Adapted from The Shack, William Young

User avatar
SilentDawning
Posts: 7348
Joined: 09 May 2010, 19:55

Re: Metrics and Agency

Post by SilentDawning » 21 Aug 2020, 07:39

nibbler wrote:
20 Aug 2020, 09:46
I want to take one step back.

It's a church, right? Why are we measuring people against metrics and holding them responsible in the first place?
I think it's important to measure results. Otherwise, you don't know how you are doing. I don't have a problem with metrics, but I do have a problem with a) coming down hard on priesthood leaders who are trying their best and b) undue emphasis on numbers without quality.

I also think it's important for the people using the numbers (the leaders above the leaders) to DO THEIR HOMEWORK before making assumptions based on the numbers. The leaders above the leaders need to ask questions about what kind of activity went on to improve the numbers. Part of this would include questions about quality.
"It doesn't have to be about the Church (church) all the time!" -- SD

"The individual has always had to struggle to keep from being overwhelmed by the tribe. No price is too high to pay for the privilege of owning yourself."

A man asked Jesus "do all roads lead to you?" Jesus responds,”most roads don’t lead anywhere, but I will travel any road to find you.” Adapted from The Shack, William Young

Minyan Man
Posts: 1999
Joined: 15 Sep 2011, 13:40

Re: Metrics and Agency

Post by Minyan Man » 21 Aug 2020, 11:01

Is it fair to hold leaders accountable for metrics that depend on others exercising their agency a certain way?

Not in a voluntary organization. Especially if I'm a volunteer in the organization.
Business organizations, schools, or other organizations that define success in dollars & cents than Yes.
I think in some situations, the church is moving away from using metrics. For example, Family History never uses measurements
for names submitted, baptisms, or number of names indexed or reviewed. They used to. You can't generate a report by
individual member. (As far as I know.)

How do you measure results FAIRLY when success of these measures depends so much on the agency of others, and not only the agency of the person/leader being held responsible for those metrics?

I use happiness, job satisfaction, extra curricular activities, friends. All of the requires have conversations with my wife, children & Grand
children.

I would never accept a position in the church if it were predicated on metrics of performance. period.

User avatar
Gerald
Posts: 447
Joined: 29 Sep 2011, 04:57

Re: Metrics and Agency

Post by Gerald » 23 Aug 2020, 08:58

Interesting topic! My academic background is research-oriented and so numbers are important to me. But we have to know what those numbers represent. It is easy to assume that the number IS the construct that we are measuring rather than an indicator of that construct. When a missionary companionship reports "five baptisms" for one month, the simplest interpretation is that five people got down into the water and went through the baptismal ceremony. But what we REALLY mean (hopefully) is that five people converted and came into the Church (and by extension) Christ. But that may or may not be the case and so the "5" becomes an indicator only of conversion. (Numbers give the illusion of accuracy and preciseness but when we're dealing with human behavior, those numbers are simply not going to have the same meaning).

I do believe that performance improves when we measure something but we have be sure that we know what we're measuring and what the numbers actually mean. The so-called "baseball baptisms" from the 50s and 60s occurred because those ceremonies were being counted (which is fine) but interpreted as being a sign of conversion (which most were not). This speaks to the evaluation that we often make of individuals (missionaries, bishops, businesspeople, teachers) for whom success is highly dependent upon the behavior of others. I think it's problematic to assume that there is a straightforward relationship between a teacher teaching a lesson and a student's score on a test of knowledge about that lesson. Absolutely, there's a relationship but it's not straightforward. The teacher has to ensure that the information is being presented but that teacher has no control over the chaotic homelife the child comes from, or the undiagnosed learning disability that the child may have, or the thoughts of the child dominated by the new computer game just purchased. Let's measure, yes, and let's hold people appropriately accountable but let's also remember that the number is not the construct and that, when talking about incredibly reactive human beings, there are many, many variables over which any teacher, bishop etc, will have no control.
Actually, in college I find that the best measure of a professor is often the reviews on ratemyprofessor. Not test scores, publications, or anything else. We could crowdsource our metrics. Have missionaries have a QR code on their pass-along cards that lets their contacts take a survey about them. Were they pushy? Polite? What is your overall impression? Did they make you more or less interested? Offer a chance to win a gift card to get people to take the survey. Ok, that might be a little too far but I think the general concept of crowdsourced metrics could be interesting. I have taken similar surveys about the quality of service at the grocery checkout line, so some organizations do use this kind of thing.
As someone whose been in and out of academia, I have to say that ratemyprofessor is another good example of a metric which is, at best, an indicator only and, at worst, horribly inaccurate. But the idea of using something similar to measure church-related concepts is certainly interesting (ratemybishop? ratemystakepresident? ratemyeldersquorumpresident?) Now THOSE evaluations would make for some interesting perusing. ;)
So through the dusk of dead, blank-legended And unremunerative years we search to get where life begins, and still we groan because we do not find the living spark where no spark ever was; and thus we die, still searching, like poor old astronomers who totter off to bed and go to sleep, to dream of untriangulated stars.
---Edwin Arlington Robinson---

NoahVail
Posts: 45
Joined: 02 Mar 2013, 22:03

Re: Metrics and Agency

Post by NoahVail » 23 Aug 2020, 09:40

For years, our ward lacked meaningful accountability. The result was that few of our youth stayed past 15. Youth would show up for an activity & were left to do whatever they came up with. We'd go years w/o a missionary. Eagle scouts became a legend from a generation ago.

Once an auxiliary leader was called, that was the end of the process. They might be pointed to a handbook that mentioned their calling. Forget about getting help assigned, no matter how many names were submitted. One time, I was so frustrated that I stapled a calling sheet to the ward roster and submitted that.

I think a better question than "Should there be accountability?" is "What should accountability look like?" We need some metrics to understand how we're doing. If those figures get used to highlight shortcomings, however, we've lost our way.

In my experience, helpful, beneficial accountability looks like a stake/ward leader asking what they can do to help, regularly.

Also in my experience: When a ward goes a generation w/o helpful accountability, members can become conditioned to the unproductive status quo. Anything else can seem like contention. Members can rise against an auxiliary leader who's trying to revive nearly dead programs. That's a terrible state to be in. To quote Pres Hinckley, "I don't recommend it".
Last edited by NoahVail on 23 Aug 2020, 10:42, edited 3 times in total.

Post Reply