MScThesisQuality

Status+background:

This is a project initiated by the D and the IT programmes together with the CSE department.
Funded by the D+IT-programmes with 200kkr for 2012 and 200kkr for 2013 (formally covering MPCSN, MPALG, MPSOF at the CSE dept. and MPIDE at the AIT department).
The CSE department aims at making the rules uniform for all the CSE-department MSc theses, including the GU side.
During 2012 the "project leader" at the CSE dept. was Nils Anders Danielsson and the PA-contact Wolfgang Ahrendt. (The 2013 project leader may not yet be decided by Miroslaw).

Reports:

CSE_MSc_reorg_proposal_version_2012-09-25.pdf
First official draft proposal (version 2012-06-03):Proposal for reorganisation of MSc theses at master's programmes associated to the D and IT programmes at Chalmers.pdf

Old information (from before the project started) is included below for reference only.

Åtgärder för att höja kvalitén på institutionens examensarbeten

Christer Carlsson, 2011-11-07

(Material in preparation for the faculty meeting on 2011-11-14. Notes are here.)

Se examensarbetet som en kurs.

Kursen startar under period 1 eller period 2 och är obligatorisk på alla masterprogram i åk 2. Kursen inleds med är en introduktion till examensarbetet och innehåller ett antal föreläsningar (och inlämningsuppgifter) som förbereder och underlättar för studenterna att genomföra examensarbetet:

klargöra syftet med och vilka krav som ställs på examensarbetet
informera om vilka bedömningskriterier som gäller för betygsättning
diskutera vad är lämpliga förslag på examensarbeten och vad är olämpliga förslag på examensarbeten
klargör handledarens roll
ge råd om hur man skriver en noggrann specifikation med tidsplan för ett examensarbete.
informera om de formella kraven på den skriftliga rapporten.
diskuterar forskningsmetoder
diskuterar vetenskaplig/ingenjörsmässigt förhållningssätt och vetenskaplig text
ge råd om hur man skriver engelsk text
ge råd hur man gör en kritiska granskningar av rapporter och föredrag

Detta inledande kursmoment måste vara avklarat innan examensarbetet får påbörjas.

Särskilj rollerna som handledare och examinator.

Handledarens roll är att ge studenten stöd, råd och kritik så att studenten kan göra ett självständigt akademiskt arbete.

Examinatorns roll är bedöma studentens arbete.

För examensarbeten som utförs i ett företag krävs att företaget tillhandahåller en ämneskompetent handledare.

Utse en huvudexaminator för varje masterprogram.

Huvudexaminatorn kan ses som kursansvarig och kan ha hjälp av en pool av handledare och eventuellt andra examinatorer.

Skärp kraven för att få påbörja examensarbetet.

I examensarbetet skall studenten, genom att utföra en självständig uppgift, visa förmåga att tillämpa de kunskaper och färdigheter som förvärvats under studietiden. För att kunna göra detta måste merparten av utbildningen vara avklarad.

Inför krav på uppnådda 250 hp för att få påbörja examensarbetet samt krav på att alla kurser från kandidatnivån är avklarade (motsvarar 70 hp inom masterutbildningen).

Inför fasta tidpunkter för att få ansöka om och få påbörja examensarbetet.

Detta innebär att olika insatser kan samordnas på ett effektivt sätt.

Skärp kraven på uppföljning av pågående examensarbeten

Inför halvtidsredovisning av hur examensarbetet framskrider enligt fastställd projektplan. Kan resultera i att examensarbetet avslutas med underkänt betyg. Inför krav på att examensarbetarna dokumenterar framskridandet i sitt arbete på sådant sätt att detta enkelt kan följas av handledare (och examinator).

Individuella upplägg för varje masterprogram.

Varje masterprogram avgör vid vilka tidpunkter ansökning och uppstart av examensarbeten sker.

Examensarbetsprocessen i korthet

Kursstart
- Obligatoriska kursmoment:
  - Vad innebär det att göra ett examensarbete?
  - Regler och krav.
  - Genomgång av bedömningskriterier.
  - Hur skall ett förslag på examensarbete utformas?
  - Hur skall rapporten utformas?
  - Forskningsmetoder och vetenskapligt förhållningssätt.
  - Kritisk granskning
  - ...
Framtagande av förslag på examensarbete
Ansökningstillfälle för förslag på examensarbeten.
Beslut om godkännande/komplettering/avslag av förslag på examensarbeten. Samt vid godkännande tillsättning av handledare och examinator.
Inledande uppstartsmöte med student, akademisk handledare och eventuell industrihandledare.
Framtagande av projektplan.
Godkännande av projektplan. (Görs av examinator och akademisk handledare. )
Genomförandet av examensarbetet.
Halvtidskontroll – hur arbetet fortskrider i förhållande till projektplanen. (* Student, handledare och examinator. Kan leda till att arbetet avslutas med ett underkänt betyg.)
Auskultation vid två presentationer av examensarbeten (Skall ske i tidigt under examensarbetet.)
Opponering på ett examensarbete. (Både muntlig och skriftlig opponering. Skall ske före den egna muntliga presentationen.)
Utkast till rapport. (Läses och kritiseras av handledaren.)
Färdig rapport. (När handledaren bedömer rapporten som fullbordad, läses rapporten av examinator.)
Fastställande av datum för muntlig presentation. (Fastställs av examinator.)
Muntlig presentation.
Eventuell korrigering av den skriftliga rapport föranledd av opponeringen. (Gör i samråd med examinator.)
Kursen Examensarbete genomförs en gång per år och är schemalagd med uppstart i period 1 eller 2 på samtliga masterprogram i åk 2.
Varje masterprogram har sitt eget introduktionstillfälle för kursen.
De obligatoriska kursmomenten kan samordnas mellan masterprogrammen.
De förslag på examensarbeten som kommer från industrin eller direkt från studenterna är ofta rudimentära, vilket gör det svårt att finna lämpliga handledare. En viktig del av kursen är att studenterna skall förstå vilka krav som ställs på ett förslag på examensarbete för att accepteras.
Ett antal fasta ansökningstillfällen finns per termin. Detta ger mycket större möjligheter till samordning vad avser tillsättning av handledare och examinator. Ger möjlighet till grupphandledning, där ett antal examensarbetare kan följa och ge kritik på varandras arbeten.
Ansökan lämnas till exjobbskoordinator, som kontrollerar förkunskapskraven.
Utvärderingen av förslaget på examensarbete görs av huvudexaminator och masterkoordinator.
Tidsplan, deadlines, former för kommunikation mellan parterna, behov kunskapsinhämtning, etc. diskuteras.
Krav skall finnas på att studenten dokumenterar framskridandet i sitt arbete på sådant sätt att detta enkelt kan följas av handledare (och examinator).

Aktörernas ansvarsfördelning

Huvudexaminator
- Godkänner/återsänder/underkänner förslaget på examensarbetet tillsammans med masterkoordinator.
- Utser handledare och eventuellt annan examinator.
Examinator
- Godkänner projektplanen tillsammans med handledaren.
- Utför tillsammans med handledare halvtidskontrollen.
- Granskar färdig rapport.
- Gör plagiatkontroll.
- Ger klartecken för muntlig presentation.
- Bedömer muntlig presentation, rapport och skriftlig opponering.
- Bedömer opponentens muntlig opponering och meddelar detta till opponentens examinator.
Handledare
- Godkänner projektplanen tillsammans med examinator.
- Utför tillsammans med examinator halvtidskontrollen.
- Granskar utkast till rapport.
- Deltar vid muntlig presentation av exjobbet.
Exjobbskoordinator
- Kontroll av behörighetskrav.
- Kontroll av att de obligatoriska momenten i exjobbskursen är avklarade.
Student
- ...

Varifrån kommer förslaget?

Högskoleverkets utvärdering av utbildningsprogrammen kommer till stor del att baseras på kvalitén på examensarbetena. Därför är kvalitén på examensarbeten en prioriterad fråga på hela Chalmers (och GU). Det konkreta förslaget som berör D&IT har jag sammanställt.

Vem ska bestämma de nya reglerna?

Institutionen bestämmer sina regler, så länge de ryms inom de centrala som finns på Chalmers (resp GU). Det är möjligt att vissa delar i förslaget passerat vad som är möjligt för institutionen att göra, t.ex höja kraven på att få påbörja examensarbetet.

Hur ser processen ut?

Vissa delar kan i förslaget kan implementeras till 2012-01-01, medan det i sin helhet inte är möjligt före 2012-08-01. IT- och D-programment har avsatt vissa projektmedel till institutionen under 2012 för åtgärder att förbättra kvalitén på examensarbetena.

Could we have Better Exjobbs?

Elad Michael Schiller 2011-10-31

(Pre-study on the MSc-thesis procedure at the CSE dept., autumn 2011. Page p,a,s,s,w,o,r,d: C,h,r,i,s,t,e,r without the commas.)

Background

Exjobb was a student internship period that facilitated their recruitment in the industry
Now, mainly for international students, can serve as a research internship platform
From 2012, exjobbs will become key evaluation criteria
- External evaluation
- Paying students expect higher standards

Existing Procedure

Find a company or an internal advisor
- Either by themselves or via a “task dispatcher”
Get their proposal approved
Eventually get their thesis approved (presentation + dissertation)

Current Achievements

An important recruitment tool
- Most civilingenjör graduates find exjobbs
- Large companies have their internal evaluation
Open also to international students
Some internal exjobbs result in a scientific publication
- We got to know some of our PhD students when they were master students

Current Problems

Student run into problems when they
- Go an a company that do not have an evaluation procedure that is equivalent to ours
- Cannot find/approve exjobbs before they start
Teachers run into problems when
- Students, that they don’t know, drop by the office
- They have wrong expectation about research‐oriented master theses

Current Problems

Master programs (are expected to) run into problems when
- Going to be evaluated by their thesis
- Paying students will start having problems

Challenges

How to keep our current achievements?
How to be prepared for external evaluation?
How to improve quality?
How to be resource smart?

Proposed Approach

Create uniform quality requirements and procedure specifications
- Define (default) requirements and allow exceptions
- Show to the student the default
- Explain the exceptions to teachers
Instead of having a complete decentralised implementation, they to involve the master program
- but just when is needed

Proposed Approach

Carry out dissemination activities of these requirements for
- Students
- Teachers
- Industry

Industrial Exjobbs

Don’t want our students to be just cheap manpower!
Default requirements: Must have a physical meeting in Chalmers with the company’s advisor, the students and Chalmers’ examiner
- Try to reach agreement
- Happens after proposal submission and before approval
Industrial exjobbs must have an internal examiner that does not advise the student; decides if everything is ok
Exceptions: Companies can publish their thesis quality requirements and procedures
Hopefully also in English
Education programs publish list of OK companies
- E.g., thesis is approved internally before starting
- Advisor must have civilingenjör/master degree
Other exceptions can be made when there is an explicit previous history of collaboration directly between the advisor and the examiner

Default Rule Separation

Advisor/referee/examiner rule separation:
- Advisor: guides the student on a weekly basis
Can recommend grade: pass, fail and borderline
- Referee: judges the dissertation if and when needed
Anonymous to the student
Short written report (no need to find all the problems)
- Examiner: decide on the recommended grade
Receives the advisor advices and looks at the dissertation
Decide if anonymous judgement is needed
Sets the final grade
Master programs can have a default examiner
For each exjobb, the default examiner can a point a well experienced dedicated examiner
In case the advisor is also well experienced, the default examiner can allow the advisor and dedicated examiner to be the same person
- But not in the case of 60 point thesis

Cutoff Dates

The department can consider two cutoff date
- Internal exjobbs only
- Say, Oct. 15 and May 15
Default examiner must deliver final decision within 14 days
This way, the division can plan the education advisor’s activity in its budget
- 4.25 master theses are 10% of position; 40h each
- 3.4 in case you are also the examiner; 50h each

Possible Extension

A dedicated teacher can help students in writing up their thesis proposal
- Cost more
- Can help the student skills
- Can detect problems early

Summary

Present my personal thoughts on how to improve how exjobbs quality and procedures
- Allow heterogeneous programs to work better
- Allow other programs to keep what they have

Other related material

Below is an inofficial copy of the 2011-05-17 version of Chalmers central guidelines for MSc thesis quality assessment. Not the final version (Johan Malmqvist).

Guidelines for assessing the quality of degree projects under Chalmers' Civilingenjör, Master of Architecture & Master of Science Programmes

Issue

Description of criteria for assessing the quality of degree projects under the Civilingenjör and Master of Architecture and Master of Science Programmes at Chalmers University of Technology.

Background

Over the past few years, Chalmers has produced and implemented common instructions for degree projects (C 2008/111). Amongst other things, the instructions describe the aim, intended learning outcomes and examination of degree projects. The instructions do not address the criteria for assessing the quality of degree projects, but there is a need for this. This is true in part for internal reasons to direct student feedback and decisions when there is a risk of a degree project not reaching the level of pass, and in part for external reasons when the quality of the degree projects forms a basis for assessment in the upcoming national system for evaluating the quality of university study programmes. In that system, universities whose students produce low-quality degree projects run the risk of losing their degree-awarding powers. On the other hand, universities whose students produce high-quality degree projects may be allocated more resources from the government. The Swedish National Agency for Higher Education will rate study programmes on a three-tier scale: inadequate quality, high quality, very high quality. This scale will also be used in these guidelines.

Aim

The guidelines aim specifically to serve as guidance when the quality of a degree project is low and a mark of pass can be called into question. The guidelines can also offer support when providing feedback to the student on his or her performance.

Structure

The guidelines are based on the intended learning outcomes for the master's degree project, the master's degree in engineering, architecture and science as formulated in Chalmers' local System of Qualifications (Reg. no. C2007/723). The criteria for inadequate quality/high quality have been formulated for all of the intended learning outcomes. The criteria for very high quality has only been formulated for certain intended learning outcomes that have been assessed as having a distinctive character – significant specialisation within the main field of study, specialised knowledge of methods, problem formulation, ability to create and evaluate new solutions, written report and independence. Inadequate quality of one or more intended learning outcomes may lead to a failing mark for the entire degree project. The examiner is responsible for making a balanced assessment. Intended learning outcomes with guidelines for quality criteria Below is a presentation of the criteria for inadequate quality (IQ), high quality (HQ) and very high quality (VHQ) for each of the intended learning outcomes for degree projects.

1. Significant specialised knowledge within the main field of study/focus of the study programme including specialised insight into relevant research and development work:

VHQ

Significant specialisation within the main field of study is demonstrated. The project utilises knowledge from studies on an advanced level within the main field of study. A comprehensive review of existing literature has been performed and there is reflection on the project's connection to the existing knowledge in the main field of study. The project contributes new knowledge in the main field of study and clearly presents it. The project demonstrates an ability to independently contribute to the field of study.

HQ

IQ

The project's connection to the main field of study is weak or non-existent. Knowledge from an advanced level within the main field of study is not utilised. A literature review and reflection on the project's connection to related areas of knowledge are missing.

2. Specialised knowledge of methods within the main field of study of the study programme:

VHQ

Potentially relevant engineering or scientific theories and methods have been identified. The selection of theory and method is well justified. Selected theories and methods have been applied in a correct and innovative manner. The project demonstrates an in-depth and broad knowledge of methods.

HQ

Potentially relevant engineering or scientific theories and methods have been identified. The selection of theory and method is well justified. The selected methods have been applied in a correct manner.

IQ

The project's selected theories and methods lack relevance. The student has not demonstrated mastery of the selected theories and methods.

3. Ability to contribute to research and development work:

HQ

The contribution to research or development work is clearly presented.

IQ

The project's character makes it difficult to connect the project to research and development work.

4. Ability to identify, formulate and manage complex problems in a critical, independent and creative manner from an overall perspective:

VHQ

The project has a clear and delimited problem or formulation of objectives. The problem/formulation of objectives has been studied in an adequate, critical and reflective manner. There is a clear connection between the problem/formulation of objectives, results, discussion and conclusions. The project's conclusions are well supported and correct.

HQ

The project has a clear and delimited formulation of the problem. The problem has been studied in an adequate manner. There is a clear connection between the problem, results and conclusions. The project's conclusions are well supported and correct.

IQ

The project does not have a clear problem or formulation of objectives or this is lacking altogether. Irrelevant method(s) used. The project does not present an answer to the problem or a result that is related to the objective. Conclusions are incorrect.

5. Ability to plan and use adequate methods to implement advanced tasks within given frameworks, as well as evaluate these efforts:

HQ

A realistic plan for the project was formulated. The set times that were communicated and established have been adhered to when performing the project. Necessary modifications for implementation have been documented and communicated.

IQ

The project has not adhered to the communicated and set times, and it has not been possible to present documentation of factors that are relevant to deviations.

6. Ability to create, analyse and critically evaluate different technical/architectural solutions

VHQ

The project produces new solutions that are analysed and evaluated in a critical manner. Alternative solutions have been produced and treated in a relevant and exhaustive manner.

HQ

The project produces solutions that are analysed and evaluated in a critical manner.

IQ

The student has not documented or presented that indicated above in a clear manner.

7. Ability to integrate knowledge in a critical and systematic manner:

VHQ

The project innovatively integrates knowledge and methods from several subjects.

HQ

Relevant knowledge and methods have been obtained and applied.

IQ

Areas that are relevant to the project are not addressed or are not used. Selected and obtained knowledge is not presented in a clear manner and is not justified.

8. Verbally and in writing, ability to clearly describe and discuss his or her conclusions in English, including the knowledge and arguments that form the basis of the conclusions:

VHQ

A very well-written report. Totality, structure and layout achieve a very high level of quality.

HQ

The project treats the selected area using relevant and correct language. Totality, structure and layout achieve a good level of quality.

IQ

The project does not make adequate use of language, which makes it difficult to understand and to provide an assessment using the report as the basis.

More detailed information is available in the guide to assessing candidates' written reports: "Bedömning av skriftlig presentation – HISS". http://thor.lib.chalmers.se/inst_fack/kandidat/Kriterier.pdf.

9. Within the framework of the specific degree project, ability to identify which problems need to be addressed to observe sustainable development:

HQ

Presents and justifies selected methods and discusses the results based on a life cycle perspective with a focus on sustainable development.

IQ

Does not observe this aspect even though the examiner assesses it to be significant for the relevant degree project.

This intended learning outcome may not be relevant for some degree projects. The examiner makes this assessment.

10. Awareness of ethical aspects related to research and development work:

HQ

Presents potential ethical consequences of the performed project.

IQ

Does not observe this aspect even though the examiner assesses it to be significant for the relevant degree project.

This intended learning outcome may not be relevant for some degree projects. The examiner makes this assessment.

11. After completing the degree project, the student shall have demonstrated the knowledge and ability required to work independently as a holder of a Civilingenjör, Master of Architecture or Master of Science degree:

VHQ

Has performed the degree project in an independent manner without the necessity of extraordinary support measures or adaptations or in any other way that required excessive resources to perform the project.

HQ

Performed the project with a reasonable amount of support.

IQ

Great need for support. The support measures have been too extensive to assume that the student would be able to work independently after receiving the degree.