pta
/
WeatherNews.jl
forked from gg/WeatherNews.jl
1
0
Fork 0
Code Pull Requests Activity
Play with data for weather waifus.
295 Commits 1 Branch 0 Tags 706 KiB
Julia 38.5%
Perl 36.8%
HTML 20.1%
Shell 4.6%
Go to file
pta be9358faf9 Include last 24 hours' schedule in video data update array 
I overlooked the fact that the limit to 12 could potentially exclude
some of yesterday's schedule from getting updated with the fetched
youtube data.
 		2024-05-27 08:06:31 -04:00
bin Add cron job to make final view count update after exactly 10 days 2024-05-09 12:46:16 -04:00
sql Replace double-quotes with single-quotes in SQL 2024-05-25 19:30:50 -07:00
src Include last 24 hours' schedule in video data update array 2024-05-27 08:06:31 -04:00
www Change header back to "Last Appearance" 2024-05-26 09:38:44 -07:00
.gitignore Send file logs to log/ directory 2023-09-23 22:36:29 -07:00
BACKSTORY.org Add a link to the spreadsheet from @smile_hakumai 2023-09-26 05:44:17 -07:00
LICENSE Initial commit 2023-07-18 05:34:50 +00:00
Project.toml Add PlotlyJS as a dependency 2024-04-29 02:39:22 -07:00
README.md Add section for git repositories in README 2024-05-26 13:24:18 -07:00
crontab Add cron job to make final view count update after exactly 10 days 2024-05-09 12:46:16 -04:00
crontab-vern Put archive directory in variable WNAR 2023-09-13 15:42:31 -07:00

README.md

WeatherNews.jl

Play with data for weather waifus.

Learning Julia

I've admired Julia from afar for too long. I want to learn how to use this language, so I started this little project to feel the language out and do something fun.

First Time

To download and compile all the depenencies, use the REPL's package mode.

  • ] (enter package mode)
  • activate .
  • instantiate
  • Hit backspace to go back to the normal Julia mode in the REPL.
julia> ]
(@v1.9) pkg> activate .
  Activating project at `~/WeatherNews.jl`

(WeatherNews) pkg> instantiate

That instantiate may take some time especially on a slower computer. After this is done, you can try playing with the code.

using WeatherNews
using WeatherNews: API, DB
v = API.video_ids()
s = WeatherNews.get_schedule()

using DataFrames
s |> DataFrame

Install yt-dlp. Optionally install jq for pretty-printing and exploring json on the command line.

Scripts

bin/weathernews.jl

This was the first script that gave birth to this project.

Running from the CLI

julia --project -O0 --compile=min bin/weathernews.jl

Running from the REPL

# ] activate .
include("bin/weathernews.jl")
main()

Data Collection System

There are a variety of scripts written in Julia, Bash, and Perl for collecting WeatherNews data. An example of how they should be run can be found in the crontab example.

  • bin/wndb-insert.jl :: This script is meant to be run from cron multiple times per day to insert new rows into the schedule table.
  • bin/wndb-video.jl :: This script is meant to be run every three hours to add video IDs and view counts to the newest WNL rows in the schedule table.
  • bin/wndb-fix-conflict.jl :: This script is meant to be run throughout the day to detect and fix schedule changes. If the API response for the schedule conflicts with what's in the database, the API response wins and what was in the database gets moved to the cancellation table.
  • bin/wndb-viewcount-update.jl :: This script is meant to be run twice during every WNL stream to make a last view count update to the stream 240 hours prior.
  • bin/schedule-archive.sh :: This script is for archiving raw JSON responses from the WeatherNews API. I'm archiving the JSON mostly for redundancy. If wndb-insert.jl were to fail, I'd hopefully have JSON data I can use to fix the database after the problem has been fixed.
  • bin/video-archive.sh :: Archive video JSON responses.
  • bin/mscale-archive.sh :: Archive mscale JSON responses.
  • bin/mscale-record.sh :: This script is meant to be run hourly to record the M-scale value from the WeatherNews API.
  • bin/aupay-insert.pl :: This script scrapes an invidious page for au PAY Market livestreams and inserts new au PAY shows it hasn't seen before. It can be run infrequently like once a week.

Web Site

Under www/ is the source for a web site written in Perl for displaying the WeatherNews data that has been collected by the various scripts.

Links

Git Repositories