30
Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim João Rocha da Silva Cristina Ribeiro [email protected] [email protected] [email protected] [email protected] Faculdade de Engenharia Universidade do Porto Faculdade de Engenharia Universidade do Porto Faculdade de Engenharia Universidade do Porto / INESC-TEC Faculdade de Engenharia Universidade do Porto / INESC-TEC

Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

  • Upload
    others

  • View
    0

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Voice recognition in the LabTablet electronic laboratory notebook

Susana Ventura Ricardo Amorim João Rocha da Silva Cristina Ribeiro

[email protected] [email protected] [email protected] [email protected]

Faculdade de Engenharia Universidade do Porto

Faculdade de Engenharia Universidade do Porto

Faculdade de Engenharia Universidade do Porto / INESC-TEC

Faculdade de Engenharia Universidade do Porto / INESC-TEC

Page 2: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Contents

1. Context and motivation

2. Managing research data with LabTablet and Dendro

a. LabTablet, an electronic laboratory notebook

b. Dendro, a collaborative data management platform

3. Speech recognition in mobile environments

4. Speech in LabTablet

5. Conclusions

2

Page 3: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Context and Motivation

3

Page 4: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Researchers know a lot about their data, so we should make it easier for researchers to describe

it adequately.

This should motivate them to do it sooner, encouraging later sharing and reuse.

4

Page 5: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Managing research data with LabTablet and Dendro

5

Page 6: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Gather

Process

Describe

Publish

Researchers Curators

InstitutionsDevelopers

Research Managers

Funders

Data Providers

http://dendro.fe.up.pt 6

Page 7: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

LabTabletan electronic laboratory notebook

7

Page 8: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

8

LabTablet

● An Electronic Laboratory Notebook (ELN)● Runs on Android devices● Allows researchers to record metadata during field runs

and others● Uses device’s onboard sensors to record metadata (GPS

location, Luminosity, Temperature, Camera…)

Page 9: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

9

LabTablet

● Metadata are represented as descriptor values● Descriptors can be generic of domain-specific

○ e.g. “Author”, “[temporal / geographical] Coverage”, “Temperature”, “Depiction”

○ They can depend on the research domain○ Researchers can be assisted in choosing which

descriptors to fill in

Page 10: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

LabTablet interacts with Dendro

● Dendro recommends descriptor sets○ Researchers fill in the descriptors during data

production○ Metadata records are pushed back to Dendro ○ Researchers then upload the data to a Dendro folder

● Experiment Metadata + Data combined

10

Page 11: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Why speech recognition?

1. Researchers are free to use their hands while they dictate to the tablet

2. Reduces the amount of interaction with the tablet to produce metadata

11

Page 12: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Dendroan ontology-based RDM platform

12

Page 13: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Screenshot taken from http://dendro-prd.fe.up.pt:3007/project/dendrorecommendation/data/Base%20Data 13

Page 14: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

14

File explorer

Page 15: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Metadata Editor

15

Page 16: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Descriptor selection area

16

Page 17: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Speech recognition in mobile environments

17

Page 18: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Speech recognition solutions

● Speech-based apps are becoming a part of daily life○ Google Now (Android)○ Siri (iOS)○ Cortana (Windows)

● Challenges○ Noisy environments○ Large amount of vocabulary

18

Page 19: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Evaluating speech recognition solutions

● Field work means that network access may be limited○ We needed offline speech recognition○ Selected library had to be open-source

● Online solutions, however, are very effective○ Faster translation speed ○ Better recognition overall

● We considered both scenarios

19

Page 20: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Online vs. Offline

● Online → Google Speech Recognition API○ Recognizes full sentences for note-taking○ Always-on speech recognition is taxing on the mobile device

■ Only active during note-taking○ LabTablet allows Portuguese and English keywords when in online

mode

20

Page 21: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Online vs. Offline (cont’d)

● Offline → CMUSphinx○ Training a speech recognition model hinders rapid prototyping○ Limited to basic word recognition

■ Keywords: “Battery” for battery temperature sensor, “luminosity” for light sensor values

○ Dictionary-based recognition■ Some very specific words are not recognized (e.g.

“descriptor”)

21

Page 22: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Speech in LabTablet

22

Page 23: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Application’s field mode

23

Page 24: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Descriptors gathering

“Descriptor”

Which descriptor?

“Description”

Save...

24

Page 25: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Online speech recognition configuration

Customization of:

● Language (ENG or PT)● Voice speed rate● Keywords

25

Page 26: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Conclusions

26

Page 27: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Conclusions

● LabTablet + Dendro

○ Tools to help researchers manage and describe data from creation to deposit

● LabTablet

○ Android-based Electronic Laboratory Notebook for researchers to use on field work or the lab

○ Uses readings from onboard sensors to fill in metadata descriptors

● Dendro

○ A web-based collaborative data management platform

○ Captures data and metadata within the research group

27

Page 28: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Conclusions (cont’d)

● Voice recognition in LabTablet

○ Voice commands for various operations

■ Record audio, take temperature/luminosity reading, record a note…

○ Uses online and offline voice recognition (Google + CMUSphinx)

● Tablet as an unobtrusive companion

○ Hands-free interaction

○ Less touch-based interactions mean less time spent handling the tablet

28

Page 29: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

Visit us athttp://dendro.fe.up.pt

Questions

Page 30: Voice recognition in the LabTablet electronic laboratory notebook · 2017. 11. 28. · Voice recognition in the LabTablet electronic laboratory notebook Susana Ventura Ricardo Amorim

This work is financed by the ERDF – European Regional Development Fund through the Operational Programme for Competitiveness and Internationalisation - COMPETE 2020 Programme, and by National Funds through the FCT – Fundação para a Ciência e a Tecnologia (Portuguese Foundation for Science and Technology) within project POCI-01-0145-FEDER-006961.

Acknowledgements

30