Play with data for weather waifus.
I've admired Julia from afar for too long. I want to learn how to use this language, so I started this little project to feel the language out and do something fun.
To download and compile all the depenencies, use the REPL's package mode.
](enter package mode)
- Hit backspace to go back to the normal Julia mode in the REPL.
cd WeatherNews.jl julia> ] (@v1.9) pkg> activate . Activating project at `~/WeatherNews.jl` (WeatherNews) pkg> instantiate
That instantiate may take some time especially on a slower computer. After this is done, you can try playing with the code.
using WeatherNews using WeatherNews: API, DB v = API.video_ids() s = WeatherNews.get_schedule() using DataFrames s |> DataFrame
This was the first script that gave birth to this project.
Running from the CLI
julia --project -O0 --compile=min bin/weathernews.jl
Running from the REPL
# ] activate . include("bin/weathernews.jl") main()
Data Collection System
There are a variety of scripts written in Julia, Bash, and Perl for collecting WeatherNews data.
An example of how they should be run can be found in the
- bin/wndb-insert.jl :: This script is meant to be run from cron multiple times per day to insert new rows into the schedule table.
- bin/wndb-video.jl :: This script is meant to be run hourly to keep the video_id of the current schedule item up-to-date.
- bin/wndb-fix-conflict.jl :: This script is meant to be run throughout the day to detect and fix schedule changes. If the API response for the schedule conflicts with what's in the database, the API response wins and what was in the database gets moved to the
- bin/schedule-archive.sh :: This script is for archiving raw JSON responses from the WeatherNews API. I'm archiving the JSON mostly for redundancy. If
wndb-insert.jlwere to fail, I'd hopefully have JSON data I can use to fix the database after the problem has been fixed.
- bin/video-archive.sh :: Archive video JSON responses.
- bin/mscale-archive.sh :: Archive mscale JSON responses.
- bin/mscale-record.sh :: This script is meant to be run hourly to record the M-scale value from the WeatherNews API.
- bin/aupay-insert.pl :: This script scrapes an invidious page for au PAY Market livestreams and inserts new au PAY shows it hasn't seen before. It can be run infrequently like once a week.
is the source for a web site written in Perl for displaying the WeatherNews data that has been collected by the various scripts.
- https://gg.vern.cc/wn/ :: Main Site
- https://wn.ydns.eu/ :: (down for maintenance)
- https://weathernews.duckdns.org/ :: (down for maintenance)
- http://sennajlwmhtxixl2qgwlvuaongxly6flffuz4roemsabjshlnnxdbbid.onion/ :: Tor Site