Steps

The 10 Easy Outcomes steps are set out below. There is a summary of these steps at the bottom of this site's front page front page and here. Not everyone will do all of these steps, and sometimes people will do them in a different order from how they are laid out here. Sometimes you will do part of a step, then do another step and come back and finish off the earlier step.

The good thing about Easy Outcomes is that if you build a good outcomes logic model according to our guidelines, you can be sure that if you are later asked to do something else, for instance - indicators for monitoring, an evaluation, an economic evaluation or even outcomes-focused contracting - you can build on the work you have already done. All you need to do is to find the appropriate step below and further develop your DoView Easy Outcomes model. 

To actually try the process yourself, download a trial version of DoView software and have a try. Once you have downloaded Doview, just download the example used here and the Easy Outcomes DoView template from the Resources page on this site and you can quickly get started setting up an Easy Outcomes DoView model. If you want to play around with some outcomes logic models from different sectors download them from the OutcomesModels.Org site. DoView makes it easier to do Easy Outcomes but, if you wish, you can do Easy Outcomes using other standard software in whatever way works for you.

Remember, you're free to use any of the material in this Easy Outcomes site for most commercial or non-commercial purposes just as long as you acknowledge this site as the source (see the Use page). 

1. Plan your Easy Outcomes work

See screenshot below

1. Plan your Easy Outcomes work

Part of planning your Easy Outcomes work is  discussing with your stakeholders who is going to be involved in doing the different stages of the Easy Outcomes analysis. Building the outcomes logic model, which is at the heart of Easy Outcomes, is best done in a small group. One approach is the following:

• Call together a meeting with your wider group of interested stakeholders and explain the Easy Outcomes process to them (use the resources on the Resources page to do this). 

• Set up a smaller group to build the outcomes logic model, do the Easy Outcomes and report back to the wider group.

The screenshot above shows the first page in the DoView model which introduces the illustrative mocked-up model which will be used to illustrate each of the Easy Outcome steps - the example of a Social Event (Party).

See all Steps     Next

2a. Build an outcomes logic model

See screenshot below

2a. Build an outcomes logic model

First you have to draw an outcomes logic model. Its called a logic model because it sets out the logical connection between steps which will get you to the outcomes for your program. When drawing it don't worry if you:

• Can't measure some of the outcomes or steps.

• Can't prove that you changed some of the outcomes or steps.*

• When you're drawing your outcomes model, use our guidelines for drawing outcomes models that will always work with Easy Outcomes.

The screenshot above shows a model of all the steps needed for a successful Social Event (A Party) mocked-up example is used to illustrate the steps. In real life examples, the model may be made up of more than one diagram like this one. In DoView you can quickly jump between them and even link steps and outcomes which are on different diagrams.  

* We'll sort all of that out later. If you want to think about it now, look at the Explanation page. 

 See all Steps     Previous   Next

2b. Check the evidence for the model

See screenshot below

2b. Check the evidence for the model

Next you can summarize any evidence you, or others, have about the connection between steps, or between steps and outcomes in your outcomes logic model. Sometimes people commission outside researchers to find what evidence exists for certain parts of their models.

The evidence can be from previous research, or from previous experience which has not been written up as research. Some links between steps in your model will be so self-evident that you will not need evidence for them. Stakeholders may have different demands about the type of evidence they will accept. The technical difficulty of proving a link also needs to be considered.  

The screenshot above shows the evidence for a set of links within the model. This evidence is stored in a DoView record associated with each link. The screenshot is part of a PDF of the model which has been created by DoView (it's not a screenshot of what you see inside DoView). The first few lines in the screenshot are the end of the listing of all links between steps in the model and then follows the list of evidence for those links in the model for which evidence has been entered.

See all Steps    Previous      Next

3a. Identify strategic priorities

See screenshot below

3a. Identify strategic priorities

Once you've drawn your outcomes model you can use it for strategic planning. You may prefer to do this step later when you've collected more monitoring and evaluation information.

• The top levels of your model will help you identify your vision and mission.

• You can work out what your priorities are in the next planning period and show them on your model.

In the screenshot above, the top level outcomes can be used to identify the overall vision as 'To create strong social networks and a good social life' and the mission as to 'To make guests and host happy'. Priorities for the next planning period can be shown by coloring them more darkly on the model. With the darkest blue the highest priorities. They are areas where there have been problems in the past and have been prioritized for more attention in the future.

 See all Steps     Previous     Next.

3b. Map current or planned activities

See screenshot below

See example below

3b. Map current or planned activities

You can map your current or planned projects or activities back onto your outcomes logic model to work out how comprehensively you are targeting outcomes.  

In the top screenshot above, are listed a set of activities which could be undertaken. One of these - Staff supervision - is highlighted. If you look at the lower screenshot you can see that this activity has been mapped* onto the steps and outcomes it is hoped it will achieve - Food prepared to adequate standard and food and drinks adequately distributed etc. The steps which it is hoped will result from the activity area marked with a link icon - an inverted V on their bottom edge.

This is very powerful, particularly in a situation where you have a number of projects (activities) undertaken by a number of separate organizations which have had their objectives specified in different ways. Building a common outcomes logic model and then mapping the projects (activities) back onto the model in this way provides a common outcomes set for the set of projects. It allows much better coordination and strategic thinking about where there may, or may not, be duplication or where more effort is needed. 

*In DoView, this mapping is done by 'cloning' a 'live copy' of the activity onto one or more DoView diagrams which contain the relevant steps and outcomes and then linking them to the steps and outcomes it is hoped it will achieve. The clone remains on the diagram in the second screenshot but could be deleted and the links will still remain. 

See all Steps     Previous     Next

4a. Put indicators onto the model

See screenshot below

4a. Put indicators onto the model 

Put any indicators you can think of onto the outcomes model next to the outcomes or steps they measure.

• Indicators are routinely collected measures of outcomes or steps.

• You may or may not be collecting these indicators at the moment. You may decide to turn ones you're not collecting at the moment into indicator projects. 

• Use the picture you now have of what indicators you are, or are planning to collect, to think carefully about whether it would be better to collect different indicators. 

In the screenshot above, the line links have been turned off so that there is space for indicators to be put on the diagram. Indicators have been put next to the steps and outcomes they are related to. If you do not visualize indicators in this way, you and your stakeholders will usually have no real idea of whether you're just collecting information on the easily measurable or on the most important steps and outcomes. 

See all Steps     Previous     Next.

4b. Identify indicators attributable to particular players

See screenshot below

4b. Identify indicators attributable to particular players 

Some indicators may be able to be attributed to (proved that they have been caused by) particular players.  

Have a look at the Explanation page if you want more information on attributable indicators.

Depending on the type of outcomes-focused contracting being used (see Step 9 below), changes in attributable indicators are usually what a player is actually contracted to deliver. Having identified attributable indicators at this stage, they can be used for discussions about contracting as part of Easy Outcomes Step 9. 

The screenshot above shows indicators which are attributable to the Waiting Staff marked with a W. There may be an interesting debate in this step as to exactly which indicators are, and are not, attributable, rather than potentially influencable by particular players*. 

For instance, those contracting the Waiting Staff may want the indicator - Response on written scale 1-5 'was the food satisfactory' to be an indicator which Waiting Staff are held to account for. However, Waiting Staff (depending on the type of outcomes-focused contracting (see Step 9 below) being used) may argue that while they potentially influence that step, they do not control it.

It all depends on whether the  Party Hosts decided to spend enough money on a sufficiently expensive menu option for the guests. Therefore, the Waiting Staff will argue that they should not be held to account for the indicator for that step, but they would usually be happy to be held to account for the indicator for lower level steps such as Food at correct temperature and Circulates around room every 5 minutes.   

*See the Explanation page for discussion of the difference.

See all Steps    Previous    Next.

4c. Identify indicator targets & success criteria

See screenshot below

4c. Identify indicator targets & success criteria

Targets and success criteria (sometimes also known as evaluative or merit criteria) are levels on quantitative indicators, or states for qualitative indicators which are used to define what constitutes achieving success.

They can be for lower steps within an outcomes model or for the highest-level of the model as a whole. Targets and success criteria are often described as being for steps, outcomes or even for a program rather than being referred to as relating to indicators. However, from a conceptual point of view they can be thought of as being specified levels on quantitative or qualitative indicators. A benchmark is a target which is set at the levels a similar program has been able to achieve on an indicator. 

Targets and success criteria will used in Step 9: Selecting outcomes-focused contracting arrangements and in Step 10: Using your model for reporting back.

• Targets - and example of a quantitative target on an indicator, using social event (party) program illustration being used here, could be something like: 'Guest responses on a written scale (1-5 'how happy are you at the present moment?') average above 4.' This target could have been developed from looking at what other social events (parties) are able to achieve and in that case it would also be a benchmark. Such targets are used later in the Easy Outcomes Step 9: Select outcomes-focused contracting arrangements.

• Success criteria can be more qualitative, but they can also use quantitative information in drawing a conclusion. An example would be that the success criteria for a Successful Party would be: 'Most people seemed to be enjoying themselves most of the time. No significant incidents which disturbed the party.' When using success criteria, particularly at the highest-levels of an outcomes model, information from both indicators and from evaluation will be taken into account in determining success.

See all Steps    Previous    Next

4d. List indicator projects

See screenshot below

4d. List indicator projects 

These are projects designed to improve one or more indicators. Sometimes they can be joint projects with other organizations which also want to set up or improve an indicator because they want to measure the same step or outcome.  

In the screenshot above you can see that under each indicator project there's a list of the indicators from the previous screenshot which that project will focus on. In DoView these are clones which are 'live copies' of the indicators which appear on the previous diagram with indicators mapped on to steps and outcomes.  

 See all Steps    Previous    Next.

5a. Put evaluation questions onto the model

See screenshot below

5a. Put evaluation questions onto the model

Put all the important evaluation questions you can think of onto your outcomes model.

• Put them next to the outcomes they focus on. This means you won't get fooled into thinking that two differently worded evaluation questions are different when in fact they are the same.* 

• Include both outcome evaluation questions (asking about your effect on high level outcomes) and non-outcome evaluation questions (asking about how to improve what you're doing or asking about how the process works). 

• Don't worry at this stage if you can, or can't, answer these evaluation questions.

• If you want to be able to do certain types of economic analysis later on you need to include the evaluation question: How much did it cost to do this activity?

In the screenshot, evaluation questions have been mapped onto your model.

*For instance, 'Did the program work?' and 'Did the program achieve its outcomes?' are usually the same question. 

 See all Steps    Previous     Next

5b. List evaluation questions going to be answered

See screenshot below

5b. List evaluation questions going to be answered

Work out which of the possible evaluation questions you came up with in the last step you're going to try to answer. One of the important differences between Easy Outcomes and most other monitoring and evaluation approaches is that it makes clear which evaluation questions you are NOT going to attempt to answer in addition to the ones you are going to try to answer. This is a very effective way of making sure that stakeholders are absolutely clear about what questions you are and are not going to be attempting. 

• Don't just assume that you have to answer high level outcome evaluation questions. Discuss with your funding or controlling organization whether or not you should be trying to answer these. Often it would be better if the funding agency evaluated them across a number of the programs they're funding.

In the screenshot each evaluation question from the previous step is discussed and a decision made about whether you're going to try to answer it.

Setting your thinking out in this was means that you can always justify your decisions to stakeholders now and in the future.

See all Steps     Previous    Next

5c. Assess possible outcome evaluation designs

See screenshot below

See screenshot below

outcomeevaluationquestionanalysis2
magnify

Click here to view screenshot

5c. Assess possible outcome evaluation 

There're only a limited number of possible high level outcome evaluation designs. Outcome evaluation asks the question - 'Can I prove that this activity actually changed high level outcomes rather than them being changed by something else'.

• Think about each of the seven possible outcome evaluation designs used in Easy Outcomes and work out whether any of them are appropriate, feasible or affordable. 

•You can find more information about these seven designs in the Easy Outcomes Workbook (PDF). If you don't know much about evaluation designs you'll have to talk to someone who does at this stage in the process. Show them your Easy Outcomes work so far and get them to help you write up just this particular step.

Keeping a clear record of your decisions about outcome evaluation is usually very helpful when outside stakeholders start asking questions about why you're doing what you're doing. The screenshot shows your conclusions about the first four of the possible outcome evaluation designs.

This screenshot shows your thinking about the other four possible high level outcome evaluation designs.

Only some of these high level outcome evaluation designs will give you the information you need to do later stages of Easy Outcomes. In particular, if you want to do certain types of economic analysis you need to have a quantitative measure of the effect of your activity on high level outcomes. This is called an 'effect size' and can usually only be worked out from doing one of the first four of these high level outcome evaluation designs.

See all Steps     Previous     Next

5d. List priority evaluation projects 

See screenshot below

5d. List priority evaluation projects

Set your evaluation projects out and under them put the evaluation questions they'll be answering.  

The screenshot shows the evaluation projects and the questions they're answering. This is very helpful in a large evaluation because it stops different evaluation sub-projects undertaken by different evaluators attempting to answer the same questions written in different ways.





See all Steps      Previous      Next

5e. Identify evaluation methods for evaluation projects

See screenshot below

See screenshot below

5e. List evaluation methods for evaluation projects

The Easy Outcomes template provides a list of possible evaluation methods (e.g. focus groups, questionnaires), timing for using the method (before, during and/or after the program) and types of analysis (e.g. statistical, qualitative etc) which you can use to answer evaluation questions within any of the evaluation projects you have listed in the previous step. 

The screenshot above shows the Easy Outcomes template slice (page) from which you can select the methods you will use for a particular project. The second screenshot shows the methods which are going to be used in in Evaluation Project 1: Party Expert Judgement Design which have been copied and pasted from the template onto this new slice (page).  

Note that the method of analysis has been included after each method and that empty boxes have been left for before and after so that the reader can very quickly see the timing of when the methods are going to be used.

In a read world example you could add additional information about the methods which on being used on this or on additional slices (pages) detailing the evaluation project methods. 

 See all Steps     Previous     Next 

6. Identify possible economic evaluation

See screenshot below

6. Identify possible economic evaluation

There're a limited number of economic evaluation designs. 

These are detailed in the Easy Outcomes Workbook (PDF).

The screenshot shows the two economic analysis designs you're going to try to do in this case. 

You can find more information about these economic evaluation designs in the Easy Outcomes Workbook (PDF). If you don't know a lot about economic evaluation you should involve someone who does in this part of your plan. 

Just show them your Easy Outcomes work so far and get them to help you write this section of your plan.

See all Steps     Previous     Next

7. Decide on piloting or full roll-out outcome evaluation

See screenshot below

7. Determine overall evaluation scheme

Work out your overall evaluation scheme. This is the overall way you are going to set up your evaluation work. In Easy Outcomes you chose between two major overall evaluation schemes. 

1. Doing outcome evaluation on the whole program

2. Just doing outcome evaluation on a pilot and then making sure that the program's applying the best practice learnt in the pilot

The screenshot shows the two different approaches to your overall monitoring and evaluation scheme. In the case of the party you would almost certainly chose the first scheme as it would cost too much to evaluate the outcomes of all of the parties that you may run in the future. After you've done the evaluation set out in this example, from then on you'd just make sure that the best practice you'd learnt from this piloting stage was being applied to future parties. This scheme is used in numerous situations where it is inappropriate, not feasible or too expensive to do outcome evaluation - for instance most of individual medical treatment uses this approach. Attributable indicators are monitored (e.g. patient attendance and sometimes treatment compliance) but no attempt is made to actually establish that any improvement in the patient can actually be attributed to the treatment (as opposed to other factors like placebo or time). Some more information on the two different schemes is available from the Systematic Outcomes Analysis site.

See all Steps     Previous    Next

8. Identify evaluation management issues

See screenshot below

8. Identify evaluation management issues

There are a number of other activities which need to be considered in undertaking an evaluation.  

The screenshot above shows how these issues are going to be dealt with in this evaluation. These issues include:

• Consultation with stakeholders on monitoring and evaluation

• Evaluation management structure

• Knowledge management for the evaluation

• Risk management

• Evaluation budget.

 See all Steps      Previous     Next

9. Select outcomes-focused contracting arrangements

See screenshot below

See screenshot below

See Screenshot below

9. Select outcomes-focused contracting arrangements 

The outcomes logic  model can be used for outcomes-focused contracting and delegation.  

In any contracting or delegation it's important that all the players are clear about who is accountable for what.  Contracts will embody targets and success criteria as set out in Step 4c: Identifying targets and success criteria. In Easy Outcomes you need to first choose between one of three possible types of contracting:

1. Contracting for attributable indicators (outputs) only.

This is the traditional outputs-focused contracting.

2. Contracting for attributable indicators (outputs) and for 'managing for outcomes'.

In the second type of contracting or delegation those contracted are also contracted to 'manage for outcomes'.* This is different from them being held accountable for achieving high-level outcomes.

3. Contracting for not fully controllable outcomes.

In the third type of contracting or delegation, those contracted are just held accountable for changing indicators of high-level outcomes. For instance, in the example being used here, the Waiting Staff would get a bonus if the guests developed strong social networks and would lose money if they did not.  

It may seem strange that people can be contracted for 'not fully controllable outcomes' in this third type of contracting. This happens in the private sector when CEOs get rewarded or punished on the basis of changes in their company's share price. When using the third type of contracting, usually not all of a person's contract payment is at stake (i.e. the will receive a bonus on top of a basic payment). People usually demand a higher reward in those times when outcomes are achieved so they can 'insure' themselves against those times when they are not. You can find more detail on all this at the Systematic Outcomes Analysis site.

Working with a visualized model helps make contracting more transparent

Using a visualized outcomes logic model such as is used in Easy Outcomes lets all the players be clear about the type of contracting or delegation arrangement and who is going to be accountable for what. The visual outcomes logic model can be dataprojected in all contracting and delegation meetings. Agreements about accountability can be noted directly onto the model as in the screenshots above. A PDF of the relevant diagrams from the DoView model can be attached as an appendix to the contract. Using this approach means that all contracting arrangements are discussed in the light of  the steps and higher-level outcomes within the model and helps maintain an outcomes-focused discussion throughout. 

The first screenshot above shows Contracting Arrangement 1, the second Contracting Arrangement 2, and the third Contracting Arrangement 3. 

*There needs to be some easy way to know whether or not someone is 'managing for outcomes'. One way is to accept that they are 'Managing for outcomes' if they have carried out an Easy Outcomes analysis and have had it successfully peer reviewed.

See all Steps     Previous     Next

10. Use your model for reporting back

See screenshot below

See screenshot below

10. Use your model for reporting back 

The final step in Easy Outcomes is  using the visualized outcomes logic model in DoView to report back on progress being made in various contexts. When doing this reporting back, the targets and success criteria developed in Step 4c: Identifying targets and success criteria will be used. Such reporting will include: 

1. Reporting back on progress on indicators of steps and outcomes. 

See the first screenshot above for an example.

2. Reporting back findings on evaluation questions.

In a subsequent strategic planning period when Easy Outcomes Step 2b. Outcomes logic model evidence is being reviewed, evidence which has been produced as a result of answering previous evaluation questions, can be summarized and put into the record associated with the evaluation project and/or the record associated with the relevant step, outcome or link. This information can be seen in the record-table when the model is dataprojected in the strategic planning meeting. Reporting back on evaluation projects in this way ensures that the 'loop is closed' in the sense of evaluation findings being fed directly into further program development and strategic thinking. This, in turn, provides the mechanism for an organization or program to operationalize and embed the concept of being a 'Learning Organization'*.

See the second screenshot above for an example. At the bottom of the screenshot is part of the record-table which shows a summary of the findings from Evaluation Project 2: Focus Groups on the music and alcohol guests would prefer.  

3. When reporting back on progress with a contract

When reporting back on a contract, an Easy Outcomes DoView model could be submitted as part of contract reporting. This could include notations of progress on contracted indicators and more detailed notes in the record-table discussing variations from what was contracted. All this information would be available in a dataprojected version of the model at all contract reporting meetings.  

Such reporting back would look like the first screenshot above. Those being contracted could note on the indicators which of the contracted indicators they wanted to discuss with those who contracted them. Details of the issues they wanted to discuss would be in the record-table.   

*A 'Learning Organization' is one which has mechanisms in place to ensure that whatever new information it finds out is fed directly into decision-making. It is often difficult to ensure this actually happens in practice and the Easy Outcomes approach can be an effective mechanism for ensuring that it does. 

See all Steps     Previous.

 Creative Commons Copyright Dr Paul Duignan 2007-2008 (most uses free)