gg
/
WeatherNews.jl
2
2
Fork 1
Code Issues 3 Pull Requests Packages Projects Releases Wiki Activity

more json data available from WNL page #2

New Issue
Closed
opened 2023-08-14 11:02:40 +00:00 by pta · 7 comments
pta commented 2023-08-14 11:02:40 +00:00
Contributor
Copy Link

I took a look at the WNL homepage source for the first time since I found the json URL. At least most of the JS that fetches dynamic information is in the script element at the bottom. Besides the timetable json URL, theres one for M-scale, and another for video IDs and thumbnail URLs of the live stream and three most recent videos.

It would be nice to record the M-scale and especially the video IDs, maybe even the thumbnail jpegs, of the live streams in the database.

You can even see their functions for translating the caster and program names in the json record.

function caster_trans(caster_name){
	var ret_name = caster_name;
	if(caster_name == "ailin"){
		ret_name = "yamagishi";
	}
	else if(caster_name == "hiyama2018"){
		ret_name = "hiyama";
	}
	else if(caster_name == "izumin"){
		ret_name = "maie";
	}
	else if(caster_name == "komaki2018"){
		ret_name = "komaki";
	}
	else if(caster_name == "matsu"){
		ret_name = "matsuyuki";
	}
	else if(caster_name == "ohshima"){
		ret_name = "oshima";
	}
	else if(caster_name == "sayane"){
		ret_name = "egawa";
	}
	else if(caster_name == "yuki"){
		ret_name = "uchida";
	}
	return ret_name;
}
function caster_kanji(caster_name){
	var ret_name = caster_name;
	if(caster_name == "yamagishi"){
		ret_name = "山岸 愛梨";
	}
	else if(caster_name == "egawa"){
		ret_name = "江川 清音";
	}
	else if(caster_name == "maie"){
		ret_name = "眞家 泉";
	}
	else if(caster_name == "matsuyuki"){
		ret_name = "松雪 彩花";
	}
	else if(caster_name == "shirai"){
		ret_name = "白井 ゆかり";
	}
	else if(caster_name == "takayama"){
		ret_name = "高山 奈々";
	}
	else if(caster_name == "hiyama"){
		ret_name = "檜山 沙耶";
	}
	else if(caster_name == "komaki"){
		ret_name = "駒木 結衣";
	}
	else if(caster_name == "uchida"){
		ret_name = "内田 侑希";
	}
	else if(caster_name == "oshima"){
		ret_name = "大島 璃音";
	}
	else if(caster_name == "tokita"){
		ret_name = "戸北 美月";
	}
	else if(caster_name == "kawabata"){
		ret_name = "川畑 玲";
	}
        else if(caster_name == "kobayashi"){
                ret_name = "小林 李衣奈";
        }
	else{
		ret_name = "";
	}
	return ret_name;
}
function title_trans(prog_title){
	var ret_title = "";
	if(prog_title == "ウェザーニュースLiVE・モーニング"){
		ret_title = "morning";
	}
	else if(prog_title == "ウェザーニュースLiVE・サンシャイン"){
		ret_title = "sunshine";
	}
	else if(prog_title == "ウェザーニュースLiVE・コーヒータイム"){
		ret_title = "coffeetime";
	}
	else if(prog_title == "ウェザーニュースLiVE・アフタヌーン"){
		ret_title = "afternoon";
	}
	else if(prog_title == "ウェザーニュースLiVE・イブニング"){
		ret_title = "evening";
	}
	else if(prog_title == "ウェザーニュースLiVE・ムーン "){
		ret_title = "moon";
	}
	return ret_title;
}

And of course their JS functions for extracting the data from the json records are there too.

I took a look at the [WNL homepage](https://onpre.weathernews.jp/wnl/) source for the first time since I found the json URL. At least most of the JS that fetches dynamic information is in the script element at the bottom. Besides the [timetable json URL](https://smtgvs.cdn.weathernews.jp/a/solive_timetable/timetable.json), theres one for [M-scale](https://weathernews.jp/mscale/json/scale.json), and another for [video IDs and thumbnail URLs of the live stream and three most recent videos](https://site.weathernews.jp/site/live/json/list.json). It would be nice to record the M-scale and especially the video IDs, maybe even the [thumbnail jpegs](https://site.weathernews.jp/app/wnlive/thumb/uN/uN41dJiD-qQ.jpg?nocache=%221691982294%22), of the live streams in the database. You can even see their functions for translating the caster and program names in the json record. ```js function caster_trans(caster_name){ var ret_name = caster_name; if(caster_name == "ailin"){ ret_name = "yamagishi"; } else if(caster_name == "hiyama2018"){ ret_name = "hiyama"; } else if(caster_name == "izumin"){ ret_name = "maie"; } else if(caster_name == "komaki2018"){ ret_name = "komaki"; } else if(caster_name == "matsu"){ ret_name = "matsuyuki"; } else if(caster_name == "ohshima"){ ret_name = "oshima"; } else if(caster_name == "sayane"){ ret_name = "egawa"; } else if(caster_name == "yuki"){ ret_name = "uchida"; } return ret_name; } function caster_kanji(caster_name){ var ret_name = caster_name; if(caster_name == "yamagishi"){ ret_name = "山岸 愛梨"; } else if(caster_name == "egawa"){ ret_name = "江川 清音"; } else if(caster_name == "maie"){ ret_name = "眞家 泉"; } else if(caster_name == "matsuyuki"){ ret_name = "松雪 彩花"; } else if(caster_name == "shirai"){ ret_name = "白井 ゆかり"; } else if(caster_name == "takayama"){ ret_name = "高山 奈々"; } else if(caster_name == "hiyama"){ ret_name = "檜山 沙耶"; } else if(caster_name == "komaki"){ ret_name = "駒木 結衣"; } else if(caster_name == "uchida"){ ret_name = "内田 侑希"; } else if(caster_name == "oshima"){ ret_name = "大島 璃音"; } else if(caster_name == "tokita"){ ret_name = "戸北 美月"; } else if(caster_name == "kawabata"){ ret_name = "川畑 玲"; } else if(caster_name == "kobayashi"){ ret_name = "小林 李衣奈"; } else{ ret_name = ""; } return ret_name; } function title_trans(prog_title){ var ret_title = ""; if(prog_title == "ウェザーニュースLiVE・モーニング"){ ret_title = "morning"; } else if(prog_title == "ウェザーニュースLiVE・サンシャイン"){ ret_title = "sunshine"; } else if(prog_title == "ウェザーニュースLiVE・コーヒータイム"){ ret_title = "coffeetime"; } else if(prog_title == "ウェザーニュースLiVE・アフタヌーン"){ ret_title = "afternoon"; } else if(prog_title == "ウェザーニュースLiVE・イブニング"){ ret_title = "evening"; } else if(prog_title == "ウェザーニュースLiVE・ムーン "){ ret_title = "moon"; } return ret_title; } ``` And of course their JS functions for extracting the data from the json records are there too.
👍 1
gg commented 2023-08-16 23:32:57 +00:00
Owner
Copy Link

This is good to know. When I get a little more free time, I'll think about recording more data.

This is good to know. When I get a little more free time, I'll think about recording more data.
gg commented 2023-08-19 09:11:14 +00:00
Owner
Copy Link

I'm just taking notes publically for later.

{
  "live": {
    "id": "pX0ZEKzxnb4",
    "thumbnail": "https://site.weathernews.jp/app/wnlive/thumb/pX/pX0ZEKzxnb4.jpg?nocache=%221692359216%22",
    "publishedAt": 1692359198,
    "title": "【LIVE】最新気象・地震情報 2023年8月19日(土)／関東から東北で局地的な激しい雷雨　夜も熱帯夜の予想 熱中症に注意〈ウェザーニュースLiVEイブニング〉",
    "duration": 0,
    "live": true
  },
  "ondemand": [
    {
      "id": "plUeoA9ujLU",
      "thumbnail": "https://site.weathernews.jp/app/wnlive/thumb/pl/plUeoA9ujLU.jpg?nocache=%221692432539%22",
      "publishedAt": 1692432539,
      "title": "お天気キャスター解説 あす 8月20日(日)の天気",
      "duration": 50,
      "live": false
    },
    {
      "id": "HUoo4GKWhvo",
      "thumbnail": "https://site.weathernews.jp/app/wnlive/thumb/HU/HUoo4GKWhvo.jpg?nocache=%220%22",
      "publishedAt": 1692431981,
      "title": "【夜の1分天気】あす2023年8月20日(日)の天気 #shorts #ウェザーニュース",
      "duration": 43,
      "live": false
    },
    {
      "id": "h0GGTEYjj6k",
      "thumbnail": "https://site.weathernews.jp/app/wnlive/thumb/h0/h0GGTEYjj6k.jpg?nocache=%221692430032%22",
      "publishedAt": 1692430112,
      "title": "【猛暑】東京都心で今年20日目の猛暑日 明日も関東では猛暑継続",
      "duration": 99,
      "live": false
    },
    {
      "id": "WeeIDZSsW_Y",
      "thumbnail": "https://site.weathernews.jp/app/wnlive/thumb/We/WeeIDZSsW_Y.jpg?nocache=%221692425746%22",
      "publishedAt": 1692425801,
      "title": "【落雷・鉄砲水・高波】夏のレジャーの注意点",
      "duration": 165,
      "live": false
    },
    {
      "id": "0IglSeFTNxA",
      "thumbnail": "https://site.weathernews.jp/app/wnlive/thumb/0I/0IglSeFTNxA.jpg?nocache=%221692420580%22",
      "publishedAt": 1692420792,
      "title": "【週間天気】来週前半は猛暑、後半は雨に ゲリラ雷雨にも注意",
      "duration": 61,
      "live": false
    }
  ]
}

We're about an hour into Evening with Airin. Here's the thumbnail.

The video id was returned as: pX0ZEKzxnb4.

The video id and thumbnail will have to be captured separately from the schedule, because they're not available until the live broadcast starts.

I also just noticed that the thumbnail url can be derived from the video id. That's one less thing I have to store in the database.

I'm just taking notes publically for later. - Asia/Tokyo 2023-08-19T17:57:53.772+09:00 - https://site.weathernews.jp/site/live/json/list.json - The response is cached by Amazon CloudFront, and cache breaking will be necessary to ensure we don't get stale data. ```json { "live": { "id": "pX0ZEKzxnb4", "thumbnail": "https://site.weathernews.jp/app/wnlive/thumb/pX/pX0ZEKzxnb4.jpg?nocache=%221692359216%22", "publishedAt": 1692359198, "title": "【LIVE】最新気象・地震情報 2023年8月19日(土)／関東から東北で局地的な激しい雷雨　夜も熱帯夜の予想 熱中症に注意〈ウェザーニュースLiVEイブニング〉", "duration": 0, "live": true }, "ondemand": [ { "id": "plUeoA9ujLU", "thumbnail": "https://site.weathernews.jp/app/wnlive/thumb/pl/plUeoA9ujLU.jpg?nocache=%221692432539%22", "publishedAt": 1692432539, "title": "お天気キャスター解説 あす 8月20日(日)の天気", "duration": 50, "live": false }, { "id": "HUoo4GKWhvo", "thumbnail": "https://site.weathernews.jp/app/wnlive/thumb/HU/HUoo4GKWhvo.jpg?nocache=%220%22", "publishedAt": 1692431981, "title": "【夜の1分天気】あす2023年8月20日(日)の天気 #shorts #ウェザーニュース", "duration": 43, "live": false }, { "id": "h0GGTEYjj6k", "thumbnail": "https://site.weathernews.jp/app/wnlive/thumb/h0/h0GGTEYjj6k.jpg?nocache=%221692430032%22", "publishedAt": 1692430112, "title": "【猛暑】東京都心で今年20日目の猛暑日 明日も関東では猛暑継続", "duration": 99, "live": false }, { "id": "WeeIDZSsW_Y", "thumbnail": "https://site.weathernews.jp/app/wnlive/thumb/We/WeeIDZSsW_Y.jpg?nocache=%221692425746%22", "publishedAt": 1692425801, "title": "【落雷・鉄砲水・高波】夏のレジャーの注意点", "duration": 165, "live": false }, { "id": "0IglSeFTNxA", "thumbnail": "https://site.weathernews.jp/app/wnlive/thumb/0I/0IglSeFTNxA.jpg?nocache=%221692420580%22", "publishedAt": 1692420792, "title": "【週間天気】来週前半は猛暑、後半は雨に ゲリラ雷雨にも注意", "duration": 61, "live": false } ] } ``` We're about an hour into Evening with Airin. Here's the thumbnail. - https://site.weathernews.jp/app/wnlive/thumb/pX/pX0ZEKzxnb4.jpg?nocache=%221692359216%22 ![airin](https://site.weathernews.jp/app/wnlive/thumb/pX/pX0ZEKzxnb4.jpg?nocache=%221692359216%22) The video id was returned as: `pX0ZEKzxnb4`. - https://youtu.be/pX0ZEKzxnb4 The video id and thumbnail will have to be captured separately from the schedule, because they're not available until the live broadcast starts. I also just noticed that the thumbnail url can be derived from the video id. That's one less thing I have to store in the database.
gg referenced this issue from a commit 2023-08-19 11:04:22 +00:00
Add a nocache helper function
gg commented 2023-08-19 12:26:56 +00:00
Owner
Copy Link

For recording purposes, I'm going to treat mscale as a value that's independent of the schedule. The unique jst column that exists in both the schedule and mscale tables can be used to join the data together.

For recording purposes, I'm going to treat mscale as a value that's independent of the schedule. The unique `jst` column that exists in both the `schedule` and `mscale` tables can be used to join the data together.
gg commented 2023-08-19 23:55:04 +00:00
Owner
Copy Link

@pta - The schedule table has a video_id column that will be updated the minute after the JSON reveals a new live video id via cron job. I currently only sync the db found on https://gg.vern.cc/wn-data/wn.db twice a day, but maybe I'll do it more often going forward. We'll see.

There's also an mscale table that records the M-scale value hourly. The schedule and mscale table can be joined using their jst column which both have.

I may redo the schedule_joined view to include this new data. It might look something like this:

SELECT seg.n      AS title,
       c.n        AS caster,
       s.video_id AS video_id,
       m.val      AS mscale,
       s.jst      AS jst
  FROM schedule s
       JOIN      segment seg ON seg.id = s.segment_id
       LEFT JOIN caster  c   ON c.id   = s.caster_id
       LEFT JOIN mscale  m   ON m.jst  = s.jst
 ORDER BY jst;
@pta - The `schedule` table has a `video_id` column that will be updated the minute after the JSON reveals a new live video id via cron job. I currently only sync the db found on https://gg.vern.cc/wn-data/wn.db twice a day, but maybe I'll do it more often going forward. We'll see. There's also an `mscale` table that records the M-scale value hourly. The `schedule` and `mscale` table can be joined using their `jst` column which both have. I may redo the `schedule_joined` view to include this new data. It might look something like this: ```sql SELECT seg.n AS title, c.n AS caster, s.video_id AS video_id, m.val AS mscale, s.jst AS jst FROM schedule s JOIN segment seg ON seg.id = s.segment_id LEFT JOIN caster c ON c.id = s.caster_id LEFT JOIN mscale m ON m.jst = s.jst ORDER BY jst; ```
pta commented 2023-08-20 17:41:19 +00:00
Author
Contributor
Copy Link

I think I should link the db in the next OP, under the schedule links, with a short description.

I think I should link the db in the next OP, under the schedule links, with a short description.
gg commented 2023-08-20 21:35:12 +00:00
Owner
Copy Link

Feel free. Eventually a site that queries the db will be made, but until that exists, a link to the raw db may suffice.

Feel free. Eventually a [site](https://warosu.org/jp/thread/43998069#p44055809) that queries the db will be made, but until that exists, a link to the raw db may suffice.
gg closed this issue 2023-08-21 16:46:41 +00:00
gg commented 2023-08-21 18:35:28 +00:00
Owner
Copy Link

Here's an unstyled dump of the schedule.
https://gg.vern.cc/wn/

Here's an unstyled dump of the schedule. https://gg.vern.cc/wn/
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: gg/WeatherNews.jl#2
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?