Higher order fused regularization for supervised learning with grouped parameters

Koh Takeuchi; Yoshinobu Kawahara; Tomoharu Iwata

doi:10.1007/978-3-319-23528-8_36

Higher order fused regularization for supervised learning with grouped parameters

Koh Takeuchi, Yoshinobu Kawahara, Tomoharu Iwata

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

4 Citations (Scopus)

Abstract

We often encounter situations in supervised learning where there exist possibly groups that consist of more than two parameters. For example, we might work on parameters that correspond to words expressing the same meaning, music pieces in the same genre, and books released in the same year. Based on such auxiliary information, we could suppose that parameters in a group have similar roles in a problem and similar values. In this paper, we propose the Higher Order Fused (HOF) regularization that can incorporate smoothness among parameters with group structures as prior knowledge in supervised learning. We define the HOF penalty as the Lovász extension of a submodular higher-order potential function, which encourages parameters in a group to take similar estimated values when used as a regularizer. Moreover, we develop an efficient network flow algorithm for calculating the proximity operator for the regularized problem. We investigate the empirical performance of the proposed algorithm by using synthetic and real-world data.

Original language	English
Title of host publication	Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2015, Proceedings
Editors	Annalisa Appice, João Gama, Vitor Santos Costa, João Gama, Alípio Jorge, Annalisa Appice, Annalisa Appice, Vitor Santos Costa, Alípio Jorge, Annalisa Appice, Pedro Pereira Rodrigues, Pedro Pereira Rodrigues, João Gama, Vitor Santos Costa, Soares Soares, Pedro Pereira Rodrigues, Soares Soares, Soares Soares, João Gama, Soares Soares, Alípio Jorge, Alípio Jorge, Pedro Pereira Rodrigues, Vitor Santos Costa
Publisher	Springer Verlag
Pages	577-593
Number of pages	17
ISBN (Print)	9783319235271, 9783319235271, 9783319235271, 9783319235271
DOIs	https://doi.org/10.1007/978-3-319-23528-8_36
Publication status	Published - 2015
Externally published	Yes
Event	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2015 - Porto, Portugal Duration: Sept 7 2015 → Sept 11 2015

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	9284
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2015
Country/Territory	Portugal
City	Porto
Period	9/7/15 → 9/11/15

All Science Journal Classification (ASJC) codes

Theoretical Computer Science
Computer Science(all)

Access to Document

10.1007/978-3-319-23528-8_36

Cite this

Takeuchi, K., Kawahara, Y., & Iwata, T. (2015). Higher order fused regularization for supervised learning with grouped parameters. In A. Appice, J. Gama, V. S. Costa, J. Gama, A. Jorge, A. Appice, A. Appice, V. S. Costa, A. Jorge, A. Appice, P. P. Rodrigues, P. P. Rodrigues, J. Gama, V. S. Costa, S. Soares, P. P. Rodrigues, S. Soares, S. Soares, J. Gama, S. Soares, A. Jorge, A. Jorge, P. P. Rodrigues, ... V. S. Costa (Eds.), Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2015, Proceedings (pp. 577-593). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9284). Springer Verlag. https://doi.org/10.1007/978-3-319-23528-8_36

Higher order fused regularization for supervised learning with grouped parameters. / Takeuchi, Koh; Kawahara, Yoshinobu; Iwata, Tomoharu.
Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2015, Proceedings. ed. / Annalisa Appice; João Gama; Vitor Santos Costa; João Gama; Alípio Jorge; Annalisa Appice; Annalisa Appice; Vitor Santos Costa; Alípio Jorge; Annalisa Appice; Pedro Pereira Rodrigues; Pedro Pereira Rodrigues; João Gama; Vitor Santos Costa; Soares Soares; Pedro Pereira Rodrigues; Soares Soares; Soares Soares; João Gama; Soares Soares; Alípio Jorge; Alípio Jorge; Pedro Pereira Rodrigues; Vitor Santos Costa. Springer Verlag, 2015. p. 577-593 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9284).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Takeuchi, K, Kawahara, Y & Iwata, T 2015, Higher order fused regularization for supervised learning with grouped parameters. in A Appice, J Gama, VS Costa, J Gama, A Jorge, A Appice, A Appice, VS Costa, A Jorge, A Appice, PP Rodrigues, PP Rodrigues, J Gama, VS Costa, S Soares, PP Rodrigues, S Soares, S Soares, J Gama, S Soares, A Jorge, A Jorge, PP Rodrigues & VS Costa (eds), Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2015, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9284, Springer Verlag, pp. 577-593, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2015, Porto, Portugal, 9/7/15. https://doi.org/10.1007/978-3-319-23528-8_36

Takeuchi K, Kawahara Y, Iwata T. Higher order fused regularization for supervised learning with grouped parameters. In Appice A, Gama J, Costa VS, Gama J, Jorge A, Appice A, Appice A, Costa VS, Jorge A, Appice A, Rodrigues PP, Rodrigues PP, Gama J, Costa VS, Soares S, Rodrigues PP, Soares S, Soares S, Gama J, Soares S, Jorge A, Jorge A, Rodrigues PP, Costa VS, editors, Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2015, Proceedings. Springer Verlag. 2015. p. 577-593. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-23528-8_36

Takeuchi, Koh ; Kawahara, Yoshinobu ; Iwata, Tomoharu. / Higher order fused regularization for supervised learning with grouped parameters. Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2015, Proceedings. editor / Annalisa Appice ; João Gama ; Vitor Santos Costa ; João Gama ; Alípio Jorge ; Annalisa Appice ; Annalisa Appice ; Vitor Santos Costa ; Alípio Jorge ; Annalisa Appice ; Pedro Pereira Rodrigues ; Pedro Pereira Rodrigues ; João Gama ; Vitor Santos Costa ; Soares Soares ; Pedro Pereira Rodrigues ; Soares Soares ; Soares Soares ; João Gama ; Soares Soares ; Alípio Jorge ; Alípio Jorge ; Pedro Pereira Rodrigues ; Vitor Santos Costa. Springer Verlag, 2015. pp. 577-593 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{03797c7013fc488f8feca5a1fcc8a01b,

title = "Higher order fused regularization for supervised learning with grouped parameters",

abstract = "We often encounter situations in supervised learning where there exist possibly groups that consist of more than two parameters. For example, we might work on parameters that correspond to words expressing the same meaning, music pieces in the same genre, and books released in the same year. Based on such auxiliary information, we could suppose that parameters in a group have similar roles in a problem and similar values. In this paper, we propose the Higher Order Fused (HOF) regularization that can incorporate smoothness among parameters with group structures as prior knowledge in supervised learning. We define the HOF penalty as the Lov{\'a}sz extension of a submodular higher-order potential function, which encourages parameters in a group to take similar estimated values when used as a regularizer. Moreover, we develop an efficient network flow algorithm for calculating the proximity operator for the regularized problem. We investigate the empirical performance of the proposed algorithm by using synthetic and real-world data.",

author = "Koh Takeuchi and Yoshinobu Kawahara and Tomoharu Iwata",

note = "Funding Information: This work was partially supported by JSPS KAKENHI Grant Numbers 14435225 and 14500801. Publisher Copyright: {\textcopyright} Springer International Publishing Switzerland 2015.; European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2015 ; Conference date: 07-09-2015 Through 11-09-2015",

year = "2015",

doi = "10.1007/978-3-319-23528-8_36",

language = "English",

isbn = "9783319235271",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "577--593",

editor = "Annalisa Appice and Jo{\~a}o Gama and Costa, {Vitor Santos} and Jo{\~a}o Gama and Al{\'i}pio Jorge and Annalisa Appice and Annalisa Appice and Costa, {Vitor Santos} and Al{\'i}pio Jorge and Annalisa Appice and Rodrigues, {Pedro Pereira} and Rodrigues, {Pedro Pereira} and Jo{\~a}o Gama and Costa, {Vitor Santos} and Soares Soares and Rodrigues, {Pedro Pereira} and Soares Soares and Soares Soares and Jo{\~a}o Gama and Soares Soares and Al{\'i}pio Jorge and Al{\'i}pio Jorge and Rodrigues, {Pedro Pereira} and Costa, {Vitor Santos}",

booktitle = "Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2015, Proceedings",

address = "Germany",

}

TY - GEN

T1 - Higher order fused regularization for supervised learning with grouped parameters

AU - Takeuchi, Koh

AU - Kawahara, Yoshinobu

AU - Iwata, Tomoharu

N1 - Funding Information: This work was partially supported by JSPS KAKENHI Grant Numbers 14435225 and 14500801. Publisher Copyright: © Springer International Publishing Switzerland 2015.

PY - 2015

Y1 - 2015

N2 - We often encounter situations in supervised learning where there exist possibly groups that consist of more than two parameters. For example, we might work on parameters that correspond to words expressing the same meaning, music pieces in the same genre, and books released in the same year. Based on such auxiliary information, we could suppose that parameters in a group have similar roles in a problem and similar values. In this paper, we propose the Higher Order Fused (HOF) regularization that can incorporate smoothness among parameters with group structures as prior knowledge in supervised learning. We define the HOF penalty as the Lovász extension of a submodular higher-order potential function, which encourages parameters in a group to take similar estimated values when used as a regularizer. Moreover, we develop an efficient network flow algorithm for calculating the proximity operator for the regularized problem. We investigate the empirical performance of the proposed algorithm by using synthetic and real-world data.

AB - We often encounter situations in supervised learning where there exist possibly groups that consist of more than two parameters. For example, we might work on parameters that correspond to words expressing the same meaning, music pieces in the same genre, and books released in the same year. Based on such auxiliary information, we could suppose that parameters in a group have similar roles in a problem and similar values. In this paper, we propose the Higher Order Fused (HOF) regularization that can incorporate smoothness among parameters with group structures as prior knowledge in supervised learning. We define the HOF penalty as the Lovász extension of a submodular higher-order potential function, which encourages parameters in a group to take similar estimated values when used as a regularizer. Moreover, we develop an efficient network flow algorithm for calculating the proximity operator for the regularized problem. We investigate the empirical performance of the proposed algorithm by using synthetic and real-world data.

UR - http://www.scopus.com/inward/record.url?scp=84984650204&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84984650204&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-23528-8_36

DO - 10.1007/978-3-319-23528-8_36

M3 - Conference contribution

AN - SCOPUS:84984650204

SN - 9783319235271

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 577

EP - 593

BT - Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2015, Proceedings

A2 - Appice, Annalisa

A2 - Gama, João

A2 - Costa, Vitor Santos

A2 - Gama, João

A2 - Jorge, Alípio

A2 - Appice, Annalisa

A2 - Costa, Vitor Santos

A2 - Jorge, Alípio

A2 - Appice, Annalisa

A2 - Rodrigues, Pedro Pereira

A2 - Gama, João

A2 - Costa, Vitor Santos

A2 - Soares, Soares

A2 - Rodrigues, Pedro Pereira

A2 - Soares, Soares

A2 - Gama, João

A2 - Soares, Soares

A2 - Jorge, Alípio

A2 - Rodrigues, Pedro Pereira

A2 - Costa, Vitor Santos

PB - Springer Verlag

T2 - European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2015

Y2 - 7 September 2015 through 11 September 2015

ER -

Higher order fused regularization for supervised learning with grouped parameters

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this