Parse json instead of HTML to retrive title and URL of the last articles. - gophercgis - Collection of gopher CGI/DCGI for geomyidae (HTM) hg clone https://bitbucket.org/iamleot/gophercgis (DIR) Log (DIR) Files (DIR) Refs (DIR) README (DIR) LICENSE --- (DIR) changeset ea8197da47392d16091d275ab2e16cebd8d71a16 (DIR) parent 621411a3bc8c7ae7e520d594581d6a16f4ff6360 (HTM) Author: Leonardo Taccari <iamleot@gmail.com> Date: Sun, 26 Aug 2018 14:08:34 Parse json instead of HTML to retrive title and URL of the last articles. XXX: Only recent articles section is implemented at the moment, the URL does XXX: not seem valid for other sections. XXX: Pagination is still not implemented ATM but it should be easier to XXX: implement now. Diffstat: internazionale/sections.dcgi | 46 ++++++++++++++----------------------------- 1 files changed, 15 insertions(+), 31 deletions(-) --- diff -r 621411a3bc8c -r ea8197da4739 internazionale/sections.dcgi --- a/internazionale/sections.dcgi Sun Aug 26 11:45:36 2018 +0200 +++ b/internazionale/sections.dcgi Sun Aug 26 14:08:34 2018 +0200 @@ -1,24 +1,17 @@ #!/bin/sh -# -# It seems that in order to enable pagination the following HTTP GET requests -# are done: -# -# <https://data.internazionale.it/stream_data/items/ultimi-articoli/0/0/$(date +'%Y-%m-%d_%H-%M-%S').json> -# -# Instead of scraping the HTML page only for the last articles this can be -# reused in order to get more data to build the DCGI and to enable -# pagination. -# - ARTICLE_CGI="/cgi/internazionale/article.cgi" section="$2" case "${section}" in - ultimi-articoli | i-piu-letti | reportage | opinioni | savagelove ) - url="https://www.internazionale.it/${section}" + ultimi-articoli) + url="https://data.internazionale.it/stream_data/items/${section}/0/0/$(date +'%Y-%m-%d_%H-%M-%S').json" + ;; + i-piu-letti | reportage | opinioni | savagelove ) + # TODO + exit 1 ;; *) exit 1 @@ -29,24 +22,15 @@ echo "Internazionale" echo "" -/usr/pkg/bin/curl -sgL "${url}" | -awk ' -/class="box-article-title"/ { - if (!match($0, /href="[^"]*"/)) { - next - } - url = substr($0, RSTART + 6, RLENGTH - 7) - url = "https://www.internazionale.it" url - - title = $0 - sub(/^ *<a href="[^"]*" class="box-article-title">/, "", title) - sub(/<\/a>.*$/, "", title) - - gsub("\\|", "\\|", url) - gsub("\\|", "\\|", title) - - printf("[0|%s|'"${ARTICLE_CGI}?"'%s|server|port]\n", title, url) -} +/usr/bin/ftp -V -o - "${url}" | +/usr/pkg/bin/jq -r ' +.items[] | ( +"[0|" + + "\(.title | gsub("\\|"; "\\|") )" + "|" + + "'"${ARTICLE_CGI}?"'" + "https://www.internazionale.it" + + "\(.url | gsub("\\|"; "\\|") )" + "|" + + "server|port]" +) ' echo ""