LTDE: A layout tree based approach for deep page data extraction

Jun Zeng; Feng Li; Brendan Flanagan; Sachio Hirokawa

doi:10.1587/transinf.2016EDP7375

LTDE: A layout tree based approach for deep page data extraction

Jun Zeng, Feng Li, Brendan Flanagan, Sachio Hirokawa

Research output: Contribution to journal › Article › peer-review

1 Citation (Scopus)

Abstract

Content extraction from deep Web pages has received great attention in recent years. However, the increasingly complicated HTML structure of Web documents makes it more difficult to recognize the data records by only analyzing the HTML source code. In this paper, we propose a method named LTDE to extract data records from a deepWeb page. Instead of analyzing the HTML source code, LTDE utilizes the visual features of data records in deep Web pages. A Web page is considered as a finite set of visual blocks. The data records are the visual blocks that have similar layout. We also propose a pattern recognizing method named layout tree to cluster the similar layout visual blocks. The weight of all clusters is calculated, and the visual blocks in the cluster that has the highest weight are chosen as the data records to be extracted. The experiment results show that LTDE has higher effectiveness and better robustness for Web data extraction compared to previous works.

Original language	English
Pages (from-to)	1067-1078
Number of pages	12
Journal	IEICE Transactions on Information and Systems
Volume	E100D
Issue number	5
DOIs	https://doi.org/10.1587/transinf.2016EDP7375
Publication status	Published - May 2017

All Science Journal Classification (ASJC) codes

Software
Hardware and Architecture
Computer Vision and Pattern Recognition
Electrical and Electronic Engineering
Artificial Intelligence

Access to Document

10.1587/transinf.2016EDP7375

Cite this

@article{d06051d959a04841bfefb0d8b0fb3708,

title = "LTDE: A layout tree based approach for deep page data extraction",

abstract = "Content extraction from deep Web pages has received great attention in recent years. However, the increasingly complicated HTML structure of Web documents makes it more difficult to recognize the data records by only analyzing the HTML source code. In this paper, we propose a method named LTDE to extract data records from a deepWeb page. Instead of analyzing the HTML source code, LTDE utilizes the visual features of data records in deep Web pages. A Web page is considered as a finite set of visual blocks. The data records are the visual blocks that have similar layout. We also propose a pattern recognizing method named layout tree to cluster the similar layout visual blocks. The weight of all clusters is calculated, and the visual blocks in the cluster that has the highest weight are chosen as the data records to be extracted. The experiment results show that LTDE has higher effectiveness and better robustness for Web data extraction compared to previous works.",

author = "Jun Zeng and Feng Li and Brendan Flanagan and Sachio Hirokawa",

note = "Publisher Copyright: {\textcopyright} 2017 The Institute of Electronics, Information and Communication Engineers.",

year = "2017",

month = may,

doi = "10.1587/transinf.2016EDP7375",

language = "English",

volume = "E100D",

pages = "1067--1078",

journal = "IEICE Transactions on Information and Systems",

issn = "0916-8532",

publisher = "Maruzen Co., Ltd/Maruzen Kabushikikaisha",

number = "5",

}

TY - JOUR

T1 - LTDE

T2 - A layout tree based approach for deep page data extraction

AU - Zeng, Jun

AU - Li, Feng

AU - Flanagan, Brendan

AU - Hirokawa, Sachio

PY - 2017/5

Y1 - 2017/5

N2 - Content extraction from deep Web pages has received great attention in recent years. However, the increasingly complicated HTML structure of Web documents makes it more difficult to recognize the data records by only analyzing the HTML source code. In this paper, we propose a method named LTDE to extract data records from a deepWeb page. Instead of analyzing the HTML source code, LTDE utilizes the visual features of data records in deep Web pages. A Web page is considered as a finite set of visual blocks. The data records are the visual blocks that have similar layout. We also propose a pattern recognizing method named layout tree to cluster the similar layout visual blocks. The weight of all clusters is calculated, and the visual blocks in the cluster that has the highest weight are chosen as the data records to be extracted. The experiment results show that LTDE has higher effectiveness and better robustness for Web data extraction compared to previous works.

AB - Content extraction from deep Web pages has received great attention in recent years. However, the increasingly complicated HTML structure of Web documents makes it more difficult to recognize the data records by only analyzing the HTML source code. In this paper, we propose a method named LTDE to extract data records from a deepWeb page. Instead of analyzing the HTML source code, LTDE utilizes the visual features of data records in deep Web pages. A Web page is considered as a finite set of visual blocks. The data records are the visual blocks that have similar layout. We also propose a pattern recognizing method named layout tree to cluster the similar layout visual blocks. The weight of all clusters is calculated, and the visual blocks in the cluster that has the highest weight are chosen as the data records to be extracted. The experiment results show that LTDE has higher effectiveness and better robustness for Web data extraction compared to previous works.

UR - http://www.scopus.com/inward/record.url?scp=85018955304&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85018955304&partnerID=8YFLogxK

U2 - 10.1587/transinf.2016EDP7375

DO - 10.1587/transinf.2016EDP7375

M3 - Article

AN - SCOPUS:85018955304

SN - 0916-8532

VL - E100D

SP - 1067

EP - 1078

JO - IEICE Transactions on Information and Systems

JF - IEICE Transactions on Information and Systems

IS - 5

ER -

LTDE: A layout tree based approach for deep page data extraction

Abstract

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this