Empirical Methods in Software Engineering - PUC-Rioinf2921/2014_2/docs/aulas/INF2921-aula2.pdf · Empirical Methods in Software Engineering Alessandro Garcia August 2014 Departamento

1

Departamento deInformática

Empirical Methods inSoftware Engineering

Alessandro Garcia

August 2014

Departamento de Informática

Website do curso

• Com todos slides e informações:

http://www.inf.puc-rio.br/~inf2921/2014_2/

– Já está disponível a partir desta semana:

• “placeholder for the slides and key references”

• agenda com aulas e datas das apresentações

2


Datas e Apresentações: Alterações

• Me enviem as datas de preferência, título e breve resumo até (no máximo) 17/09

• Apresentações do Tema do Trabalho– 24/09 e 8/10

– 24/09: Diego, Leonardo, Benjamin? Ok? Mais alguém? Trocas?

– Apresentação: 25 mins + 10 mins perguntas/sugestões

• Não haverá aula em 01/10– reposição na segunda parte de outubro

• Resultados dos Estudos Empíricos– Datas: 26 de novembro e 3 de dezembro

• Se o estudo empírico for em grupo, espera-se que todos os membros contribuam ao longo das apresentações


Apresentação do Plano do Estudo

• Me enviem um e-mail até 17 de setembro, informando:– individual ou em dupla

– descrição do tema do estudo• título inicial

• qual o objetivo?

• você pode se juntar à um colega que já tem uma boa idéia:

– da aplicação à ser utilizada, e/ou

– dos objetivos do estudo

– fiquem à vontade para discutir idéias comigo

– Apresentação inicial: aprox. 25 mins + 5 mins perguntas

3


Previous: Experimental Software Engineering

• Why?

– empirical studies in software engineering (SE) supports

• the advancement of the field through experimental processes

– common wisdom, intuition, speculation and proofs of concepts

are not reliable sources of credible knowledge

• transfer of research results to mainstream software

development requires empirical evidence

• What?

– experimental software engineering describes the methods and

techniques for performing empirical studies

• the process supports us in doing our studies correctly


This Lecture…

• Brief overview

– Types of Empirical Methods

• using concrete examples

– studies developed as coursework in the past in this

course

Victor R. Basili, Richard W. Selby, David H. Hutchens: Experimentation in

Software Engineering. IEEE Trans. Software Eng. 12(7): 733-743 (1986)

Barbara A. Kitchenham, Shari Lawrence Pfleeger: Principles of survey

research part 6: data analysis. ACM SIGSOFT Software Engineering Notes

28(2): 24-27 (2003)

4


1st Type: Case Study

• Observational, often in-situ for an ongoing project

– E.g. assess the effectiveness of RUP to promote overall design quality in company X and/or Y

– E.g. does the use of the product line approach accelerates time to market of Web information systems in company X? in Y?

• e.g. effort to instantiate a produt is lower?

• Study a phenomenon in a specific space and time period

– Applicable to dynamic or larger studies:

• Long term evaluations

• Industrial evaluations

– Can be also used to compare approaches

(e.g. on a project and sister project)

• not an experiment because:– subjects are not randomly selected

– little or no control over the experience of developers, tools employed, etc...

– Easier to plan than controlled experiments


Case Study

• Harder to control, hence less useful for asserting causality

5


Case Study

• Harder to control, hence less useful for asserting causality

Quais outras variáveis não controladas podem influenciar as

diferenças de resultados mais decisivamente?


Case Study - Example

• Case: effect of using new modelling concepts when specifying aspects of test management

6


Types of Empirical Strategies

• From Case Studies to Experiments:


Experiments

• Most controlled form of study, laboratory, manipulation of

variables:

– E.g. time spent to develop and test a few auxiliary functions using:

• pair programming vs. solo programming

• testing last vs. testing first

– Controlled environment, where it is possible/doable to control:

• selection of subjects (e.g. similar backgrounds)

• learning curve

• application to both novices (students) and industry

programmers

• randomization over subjects

7


Continuous Modularity

• support: warn and provide feedback to developers when editing the code

13

Before After

Ongoing

Continuously

Inbetween

Programmer

editing code

specifying

designing

compiling

committing

maintaining

evolving


Continuous Modularity Support

14

Did I violate any

design rule?

How is the

modularity ‘health’

of my code?

Will I introduce

merge conflicts?

Did I introduce security

vulnerabilities?

How should I

refactor my code?

How to implement

this ‘catch’ block?

Is my list of assertions

ok?

8


Claim: contextualized support...

15

m1(){

c0();

c1();

…

ci(); // throws T1

…

cn();

}

... for instance, to help coding exception handling

IDEs (e.g. Eclipse) only provide bad recommendations

How to implement

my catch block

here?


IDE view with extra-information and code-examples provided by the heuristics

JimboEH: supporting programmers of exception handling code

Continuous Robust Programming

9


Continuous Code Quality

17

[Murphy-hill, 2013] Emerson R. Murphy-Hill, Titus

Barik, Andrew P. Black: Interactive ambient visualizations for soft advice. Information Visualization 12(2):

107-132 (2013)

• What is it?

interactive support for

modularity awareness

• Stench Blossom

(flor de cheiro)


Continuous Code Anomaly Detection

• What is it? Active and Explanation Views...

18

Which anomaly?

Which elementscomprise a anomaly?

10



19

Did I violate any

design rule?

How is the


of my code?

Will I introduce

merge conflicts?


vulnerabilities?

How should I

refactor my code?

How to implement



ok?

How to perform a controlled experiment in this context?

Possible?



20

Did I violate any

design rule?

How is the


of my code?

Will I introduce

merge conflicts?


vulnerabilities?

How should I

refactor my code?

How to implement



ok?Continous Modularity vs.

Convention Modularity

Impossible to perform an controlled all-encompassing EXPERIMENT in this context

11



21

Did I violate any

design rule?

How is the


of my code?

Will I introduce

merge conflicts?


vulnerabilities?

How should I

refactor my code?

How to implement



ok?

Impossible to perform an controlled all-encompassing EXPERIMENT in this context:

too many variables to understand and control

e.g. levels of expertise of programer addressing each of

these questions; too many complementary tools to each

task; etc..


Solution: choose a specific controllable context (and question)

22

Did I violate any

design rule?

How is the


of my code?

Will I introduce

merge conflicts?


vulnerabilities?

How should I

refactor my code?

How to implement



ok?

12


Much higher identification of anomalies

ContinuousDetection

ConventionalDetection

Controle: nível de

experiência across

types of subjects

and within subjects


Initial evidence of higher refactoring quality

• As developers get feedback when they are working in that

context

24

Continuous Detection Conventional Detection

Bad

refactoring

Appropriate

refactoring

13


ContinuousDetection


Controle: nível de

experiência across

types of subjects

and within subjects

Uncontrolled candidate variable that

might have influenced the main

results?


ContinuousDetection


Controle: nível de

experiência across

types of subjects

and within subjects

Learning curve

• always same application or one always the same technique after

another

14


Experiments

• Example of how to control subjects’ experience and learning curve

• Comparing three variability management techniques: from artifact comprehension point of view

– GenArch+, PureVariants, CIDE (similar to FeatureMapper)


Experiments

• Most controlled form of study, laboratory, manipulation of

variables:

– Reproducible

• Because they demand less time in the execution

• The artefacts are produced just for the experiment

– … and can be reused (“as they are”) in experiment repetitions

– Validate more specific claims

• E.g. evaluate time spent to change a piece of module using

specific variability mechanisms

– conditional compilation vs. AOP

• E.g. efficacy of the metrics to detect a bad smell in a certain

design

15


Experiments

• Another example:

– comparing the effort of using different model

composition techniques


Experiments

• Example of applying the heuristic-based and specification-based techniques

Heuristics-based

Specification-

based

16


Experiments

• The experiment was conducted with

– 16 subjects were professionals

– 8 subjects were students with professional experience


Experiments

• The process…

17


Intuition

RSA

Epsilon

Effort Correctness

L

H

L

H


Results: Effort

RQ1: Which approach requires less effort to apply?

Med

ian

of

Eff

ort

(m

in) Total Effort

18


Experimental Results: Effort

RQ1: Which approach requires less effort to apply?

• Effort

Epsilon required more effort than IBM RSA

Epsilon required more effort than merge/override

Total Effort


Experiments

• Resultados:

19


Difficulties on conducting this experiment

• Training was too expensive and needs to be carefully designed

– it took one day in each institution + preparation

– examples chosen cannot influence the participants in

favour of one technique

• Most professional developers were already used to apply RSA mechanisms

– how to take them to a similar level of knowledge with

the use of Epsilon and manual heuristics?

– we needed to apply questionnaire/exercises to check

their level of knowledge before the experiment itself


Difficulties on conducting this experiment

• While the heuristics available are essentially the same…

– the DSLs for specification-based composition are very

different

• E.g. Epsilon against MATA

– therefore, we could not claim the results could be

generalized

– however, we can only use stable tools in experiments

• we are measuring time/correctness

20

Departamento deInformática

Empirical Methods inSoftware Engineering

Alessandro Garcia

August 2014

Documents

Empirical Methods in Software Engineering - PUC-Rioinf2921/2014_2/docs/aulas/INF2921-aula2.pdf · Empirical Methods in Software Engineering Alessandro Garcia August 2014 Departamento