gg
/
WeatherNews.jl
2
2
Fork 1
Code Issues 5 Pull Requests Packages Projects Releases Wiki Activity

au PAY videos should be recorded in the database #5

New Issue
Closed
opened 2023-09-02 16:32:44 +00:00 by gg · 14 comments
gg commented 2023-09-02 16:32:44 +00:00
Owner
Copy Link

Initial Proposal

Database

  • Fortunately, the data can be recorded without changing the current schema.
    • A new row could be added to the segment table for au PAY Market shows.
    • The au PAY shows could then be recorded in the schedule table normally.
    • The schedule.video_id field could contain the numeric id that they use, and a variety of URLs can be generated based on that.

Template

  • Link generation is the main blocker right now.
    • The current template and supporting code assumes schedule.video_id is a YouTube video_id.
    • The introduction of au PAY videos would break that assumption.
    • It wouldn't break the site, but it would generate non-sensical links until a way to handle link generation correctly is made.
  • I'm also not totally happy with the templating library I chose, and I want to swap it out before I get too far and it becomes too painful to change.
## Initial Proposal - [>>>/jp/44500927](https://boards.4channel.org/jp/thread/44465230#p44500927) - [>>>/jp/44500927](https://warosu.org/jp/thread/44465230#p44500927) (archive) ## Database - Fortunately, the data can be recorded without changing the current schema. + A new row could be added to the `segment` table for au PAY Market shows. + The au PAY shows could then be recorded in the `schedule` table normally. + The `schedule.video_id` field could contain the numeric id that they use, and a variety of URLs can be generated based on that. ## Template - Link generation is the main blocker right now. + The current template and supporting code assumes `schedule.video_id` is a YouTube video_id. + The introduction of au PAY videos would break that assumption. + It wouldn't break the site, but it would generate non-sensical links until a way to handle link generation correctly is made. - **I'm also not totally happy with the templating library I chose, and I want to swap it out before I get too far and it becomes too painful to change.**
gg commented 2023-09-02 16:56:33 +00:00
Author
Owner
Copy Link

Strangely, the page for id 10211 doesn't show a video.
https://wowma.jp/event/live-tv/play.html?live_id=10211

I can confirm that the m3u8 link shows Ayaka though.
https://vod.check.tv/lives/10211/360p/index.m3u8

Strangely, the page for id 10211 doesn't show a video. https://wowma.jp/event/live-tv/play.html?live_id=10211 I can confirm that the m3u8 link shows Ayaka though. https://vod.check.tv/lives/10211/360p/index.m3u8
Screen Shot 2023-09-02 at 09.53.44.png
1.8 MiB
index.m3u8.00:00:06.733.jpg
67 KiB
Screen Shot 2023-09-02 at 09.53.44.png index.m3u8.00:00:06.733.jpg
pta commented 2023-09-02 17:01:29 +00:00
Contributor
Copy Link

Once that link generation is fixed, maybe you could have links to open the m3u8 streams in a video player in browser to make it easy for non mpv-users to watch.

Once that link generation is fixed, maybe you could have links to open the m3u8 streams in a video player in browser to make it easy for non mpv-users to watch.
gg commented 2023-09-02 17:06:00 +00:00
Author
Owner
Copy Link

Thumbnails are easy to derive.
>>>/jp/44501949
https://image.check.tv/lives/10213/vertical.jpg
https://image.check.tv/lives/10211/vertical.jpg

Thumbnails are easy to derive. [>>>/jp/44501949](https://warosu.org/jp/thread/44465230#p44502205) https://image.check.tv/lives/10213/vertical.jpg https://image.check.tv/lives/10211/vertical.jpg
gg commented 2023-09-02 17:08:12 +00:00
Author
Owner
Copy Link

Once that link generation is fixed, maybe you could have links to open the m3u8 streams in a video player in browser to make it easy for non mpv-users to watch.

Do you have any JavaScript video player recommendations? I don't know what's good these days. Oh wait. I can just use the <video> tag. Never-fucking-mind.

> Once that link generation is fixed, maybe you could have links to open the m3u8 streams in a video player in browser to make it easy for non mpv-users to watch. Do you have any JavaScript video player recommendations? I don't know what's good these days. Oh wait. I can just use the `<video>` tag. Never-fucking-mind.
pta commented 2023-09-02 17:20:14 +00:00
Contributor
Copy Link

Do you have any JavaScript video player recommendations? I don't know what's good these days.

lul I was going to suggest looking at the video players aqstream and another free foreign tv site uses. I remember one of them is GPL'd, but I don't remember whether it provides features over the <video> tag.

> Do you have any JavaScript video player recommendations? I don't know what's good these days. lul I was going to suggest looking at the video players aqstream and another free foreign tv site uses. I remember one of them is GPL'd, but I don't remember whether it provides features over the `<video>` tag.
gg commented 2023-09-02 18:30:20 +00:00
Author
Owner
Copy Link

image

I put one au PAY entry into the schedule to see how it looks.

![image](/attachments/11426513-8230-4fce-927a-0634462454c4) I put one au PAY entry into the schedule to see how it looks.
image.png
37 KiB
gg commented 2023-09-03 03:29:22 +00:00
Author
Owner
Copy Link

Depending on browser and platform, the <video> tag may not be able to handle streams on its own. It seems phones are often more capable by default there. However, there are libraries that can fill in the gaps.
https://github.com/videojs/http-streaming
https://github.com/videojs/video.js

Depending on browser and platform, the `<video>` tag may not be able to handle streams on its own. It seems phones are often more capable by default there. However, there are libraries that can fill in the gaps. https://github.com/videojs/http-streaming https://github.com/videojs/video.js
gg commented 2023-09-03 05:14:53 +00:00
Author
Owner
Copy Link

@pta - The links are busted, but I got everything I could in the database using the new bin/aupay-insert.pl script. I forgot that I was trying to learn Julia and ended up doing it in Perl.
https://gg.vern.cc/wn/segment/?n=au+PAY

@pta - The links are busted, but I got everything I could in the database using the new `bin/aupay-insert.pl` script. I forgot that I was trying to learn Julia and ended up doing it in Perl. https://gg.vern.cc/wn/segment/?n=au+PAY
gg referenced this issue from a commit 2023-09-03 07:18:33 +00:00
Fix the aupay links #5
gg commented 2023-09-03 08:00:42 +00:00
Author
Owner
Copy Link

The Mystery of 10211

INSERT INTO schedule (segment_id, caster_id, video_id, jst)
VALUES (8, 6, "10211", "2023-07-27T18:00:00+09:00");
# The Mystery of `10211` - I watched a little bit of it to try to figure out when it happened, and the hashtag they used for the show was [`#あやちとプレイバック`](https://twitter.com/search?q=%23%E3%81%82%E3%82%84%E3%81%A1%E3%81%A8%E3%83%97%E3%83%AC%E3%82%A4%E3%83%90%E3%83%83%E3%82%AF). - The show was broadcast on `2023-07-27T18:00:00+09:00`. - A few official tweets: - https://twitter.com/livetv_aupaymkt/status/1683784219288539136 - https://twitter.com/ayaka196196/status/1684485663792812034 - Unlike many of the other au PAY shows, this one didn't seem to have a preshow on YouTube. - That's how I found most of the other au PAY shows. - The bitly URL is https://bit.ly/3KAe8xh and it does hide the video_id 10211 behind its redirect. ```sql INSERT INTO schedule (segment_id, caster_id, video_id, jst) VALUES (8, 6, "10211", "2023-07-27T18:00:00+09:00"); ```
pta commented 2023-09-03 15:14:33 +00:00
Contributor
Copy Link

I was wondering where Julia was going to be used for the website. The Haskell library I was trying out the other day throws type errors in ghci if you try to generate invalid html, like putting a <p> in the <head>. The parser Julia libraries you linked look good.
The m3u8 and thumbnail links work. I guess you'll eventually make the "vid" links open a reliable in-browser player.
Since Nana didn't do any pre-stream youtube streams, I'm eager to find those manually. But first, I need to write a little shell function wrapper for graphicsmagick to combine kindle book page fragments into full-page images so I can easily post gravure from Yui and other casters' photobooks.
The ranking page should note the day you started counting appearances.

I was wondering where Julia was going to be used for the website. The Haskell library I was trying out the other day throws type errors in ghci if you try to generate invalid html, like putting a `<p>` in the `<head>`. The parser Julia libraries you linked look good. The m3u8 and thumbnail links work. I guess you'll eventually make the "vid" links open a reliable in-browser player. Since Nana didn't do any pre-stream youtube streams, I'm eager to find those manually. But first, I need to write a little shell function wrapper for graphicsmagick to combine kindle book page fragments into full-page images so I can easily post gravure from Yui and other casters' photobooks. The ranking page should note the day you started counting appearances.
gg commented 2023-09-03 17:07:47 +00:00
Author
Owner
Copy Link

I was wondering where Julia was going to be used for the website.

I can't get Julia to run at all on vern.cc -- not even a REPL. The nix packaged version refuses to install, because the package is broken. Also the packaged version is really old. I then tried downloading the latest binaries, and they won't run either. It's weird.

I think I'm just going to keep the web site as little Perl CGI scripts.

  • I've figured out how to deploy my Perl code to their nix system. (It's a bit ugly, but it works.)
  • I'm not sure I want to deal with the hassle of
    • setting up a persistent server,
    • setting it up as a user-level service under nix, and
    • putting process monitoring on it.
  • It might be nice to know eventually, but not in the mood for that now.

The Haskell library I was trying out the other day throws type errors in ghci if you try to generate invalid html, like putting a <p> in the <head>.

They love correctness.

The parser Julia libraries you linked look good.

Maybe the next time I need to scrape, I'll try them. I didn't even think to look for Julia libraries until I was mostly done with the Perl-based scraping solution.

The m3u8 and thumbnail links work. I guess you'll eventually make the "vid" links open a reliable in-browser player.

When I wrote that initial comment, they were still broken YouTube-based links. I was thinking of calling it a night, but I couldn't leave those links totally broken. I still want to swap out the templating system, so I don't want to mess with the existing templates too much more.

Since Nana didn't do any pre-stream youtube streams, I'm eager to find those manually. But first, I need to write a little shell function wrapper for graphicsmagick to combine kindle book page fragments into full-page images so I can easily post gravure from Yui and other casters' photobooks.

It should be a fun little project. I think you've mentioned this a few threads ago.

The ranking page should note the day you started counting appearances.

Eventually, I'd like to have monthly and yearly breakdowns, but right now, it's scanning the whole schedule table.

> I was wondering where Julia was going to be used for the website. I can't get Julia to run at all on vern.cc -- not even a REPL. The nix packaged version refuses to install, because the package is broken. Also the packaged version is really old. I then tried downloading the latest binaries, and they won't run either. It's weird. I think I'm just going to keep the web site as little Perl CGI scripts. - I've figured out how to deploy my Perl code to their nix system. (It's a bit ugly, but it works.) - I'm not sure I want to deal with the hassle of - setting up a persistent server, - setting it up as a user-level service under nix, and - putting process monitoring on it. - It might be nice to know eventually, but not in the mood for that now. > The Haskell library I was trying out the other day throws type errors in ghci if you try to generate invalid html, like putting a `<p>` in the `<head>`. They love correctness. > The parser Julia libraries you linked look good. Maybe the next time I need to scrape, I'll try them. I didn't even think to look for Julia libraries until I was mostly done with the Perl-based scraping solution. > The m3u8 and thumbnail links work. I guess you'll eventually make the "vid" links open a reliable in-browser player. When I wrote that initial comment, they were still broken YouTube-based links. I was thinking of calling it a night, but I couldn't leave those links totally broken. I still want to swap out the templating system, so I don't want to mess with the existing templates too much more. > Since Nana didn't do any pre-stream youtube streams, I'm eager to find those manually. But first, I need to write a little shell function wrapper for graphicsmagick to combine kindle book page fragments into full-page images so I can easily post gravure from Yui and other casters' photobooks. It should be a fun little project. I think you've mentioned this a few threads ago. > The ranking page should note the day you started counting appearances. Eventually, I'd like to have monthly and yearly breakdowns, but right now, it's scanning the whole schedule table.
gg commented 2023-09-04 12:10:30 +00:00
Author
Owner
Copy Link

Cockblocked by CORS?

My first attempt at trying to use video.js yielded this error message.

Access to XMLHttpRequest at 'https://vod.check.tv/lives/10213/1080p/index.m3u8' from origin 'http://localhost:4444' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

What's funny is that if I use a plain old <video> tag without video.js, it works on my phone but not on the desktop. The video tag on the phone is more capable somehow.

## Cockblocked by CORS? My first attempt at trying to use video.js yielded this error message. ``` Access to XMLHttpRequest at 'https://vod.check.tv/lives/10213/1080p/index.m3u8' from origin 'http://localhost:4444' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. ``` What's funny is that if I use a plain old `<video>` tag without video.js, it works on my phone but not on the desktop. The video tag on the phone is more capable somehow.
pta commented 2023-09-05 01:33:09 +00:00
Contributor
Copy Link

This turns out to be harder than I thought. I just looked at some old stackoverflow threads about playing m3u8 with the video tag and they all conclude that just a few browsers support it. The video tag was only really meant to play mp4, webm, ogg, etc. Not HLS or whatever.
The first answer to this stackoverflow thread talks about the CORS thing. I don't quite get it, but I seems like work that vern would have to do on the webserver. You might just have to stick to the phone-capable links. Desktop users should be using mpv or vlc anyways.

This turns out to be harder than I thought. I just looked at some old stackoverflow threads about playing m3u8 with the video tag and they all conclude that just a few browsers support it. The video tag was only really meant to play mp4, webm, ogg, etc. Not HLS or whatever. The first answer to [this stackoverflow thread](https://stackoverflow.com/questions/29351225/playing-with-video-js-ustream-m3u8-file-streaming) talks about the CORS thing. I don't quite get it, but I seems like work that vern would have to do on the webserver. You might just have to stick to the phone-capable links. Desktop users should be using mpv or vlc anyways.
👍 1
gg commented 2023-09-05 03:28:57 +00:00
Author
Owner
Copy Link

I'm going to consider this issue closed for now.

  • au PAY shows can now be added in an automated way if they have a YouTube livestream before the main au PAY stream.
    • If the script fails, manual insertions aren't too bad since they're infrequent.
    • We have usable m3u8 links that can be played which is what matters most.
      • Quite a few of these archived videos are hidden on the mobile site, but we have access to them.
  • We did a little extra credit video research.
    • Maybe we revisit this later, but no rush.

That was fun.

I'm going to consider this issue closed for now. - au PAY shows can now be added in an automated way if they have a YouTube livestream before the main au PAY stream. - If the script fails, manual insertions aren't too bad since they're infrequent. - We have usable m3u8 links that can be played which is what matters most. - Quite a few of these archived videos are hidden on the mobile site, but we have access to them. - We did a little extra credit video research. - Maybe we revisit this later, but no rush. That was fun.
gg closed this issue 2023-09-05 03:28:58 +00:00
gg referenced this issue from a commit 2023-09-06 15:21:49 +00:00
Insert Nana's au PAY show #5
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: gg/WeatherNews.jl#5
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?