Big Book of Study Stuffs

To get Prime-san's document indexed by search engines, here's a couple of really simple ideas: (1) set something up that, after every change made to the document, it's exported via Google Drive API to PDF and/or HTML and placed on the Kotoba Miners website. (2) Translate this exported file to MediaWiki. Nothing really beats Google Drive for collaborative editing (and you're lucky to have a community here that isn't fanatical about this stuff , so let that be the "pristine source".

First: we have that book! Second: without getting it out of the bookshelf or logging into YesJapan, just looking at the images there, they don't look much like Prime-san's document or your CourseBase notes, which summarize the Kotoba Miners course syllabus. (Though I do have a suggestion for Prime-san: indicate in the section headings what building & floor a topic is tentatively discussed on, i.e., JP2.5?) Sorry for being dense, I guess I'm asking for more details on what you envision translating this Big Book of Study Stuffs or the CourseBase notes to.

How do you mean quizy/flashcard-ish? People descibe Anki usually as a flashcard application or do you mean, something with less SRS?

Please don't let my gasbagging distract you from rolling something of your own! My projects are very scatter-brained. The gap I see in offerings like Memrise/Anki/Mnemosyne/Duolingo is that they are primarily SRS-focused (~90%) with very poor content management (~10%; DuoLingo of course has none). I want to reverse that. The app I am dreaming of will instead focus on content management: I want it to be easy to make not just flashcards/wordlists but also enroll ebooks, subtitles, audio and video, etc., into my personal learning database, to browse and search this database, and in addition to all this, set up prompts to get quizzed on portions of it.

Of course, this description is very abstract because it was developed from a couple of specific use cases. On LINE I got some good feedback (thanks especially @Nayelianne) on a hypothetical system where you'd paste a big block of Japanese text you wanted to learn, and after reviewing the words/kanji you already knew, the system would start teaching you kanji that you didn't already know, and as soon as you learned enough kanji to learn a word, it taught you the word, and then as soon as you knew all the words in a sentence, it quizzed you on that sentence cloze-deleted (fill-in-the-blank), and proceed in this way till you theoretically learned everything in the excerpt.

An aside: this is obviously very different than RTK. I know cheap-sensei loves RTK, and I have gone thru 3/4ths of it myself and love kanji and Heisig's technique of learning kanji in primitive-based order, but I've grown dissatisfied with being prompted with English keywords (even after I heavily annotate them to personalize them for myself) to produce a written kanji. I mean, look: I have tools like MeCab and JDepP (a bunsetsu chunker & dependency parser) that can chop up Japanese sentences with quite good accuracy; I have corpora that let me build good word/kanji frequency tables and find common collocations; I have a ton of anime and manga and literature that I'm deeply familiar with in translation/subtitle which could give me a strong-as-steel contextual embedding for anything I learn---in short, this isn't the 80s!, and we have all the tools we need to heavily personalize a kanji/vocab-learning curriculum for each person, based on their interests and data. ReadTheKanji, Powerlevel Japanese, FeedMeJapanese, etc., (NOT the horrible Core* decks in Anki) are all pointing towards this highly personalized, multimedia contextualization of what you're learning and being quizzed on.

A concrete example: I have Japanese subtitles for a Ghibli film that I've watched countless times with my family. Anki's sub2srs plugin sounds great, but Anki is so focused on SRS that the experience of casually browsing and editing cards is hamstrung. I'm dreaming of an app that'll accept the film and subs as uploads, parses the text data (subs), teaches me the kanji that I don't yet know, then the words, and finally quizzes me on sentences via fill-in-the-blanks (including particles and conjugations and collocations!), with the audio and a screenshot/animated GIF from the film constantly available to serve as the contextual concrete to embed my memories in. It'll quiz me more regularly on words that are more common, until I'm good with those (which'll happen quickly since they are probably going to be frequently used in the film), then focus on words that are less common. So while it may teach me what a rare kanji is and how to draw it, it won't ask me to "review" them right away with the same frequency as more "important" kanji. I expect to interact with this database, including my annotations and mnemonics, much more often than quizzes are scheduled.

Note that while I've not mentioned SRS, it's a big part of the background, and it'll be smart: informed by frequencies and know about collocations. And there's nothing here about "minimize the amount of time you spend reviewing" which is a part of Anki's philosophy. I want this app to be something that I want to keep open all the time, because I can so easily add more content (ebooks, manga with "subtitles" though this is hard work even with KanjiTomo, music with lyrics, etc.) and have a backlog of content I'm interested in studying to turn to. Eventually I'd like to be able to share the content I've added with others, so they don't have to transcribe manga themselves (I'm slowly working on transcribing Yotsuba!). Or make "packs" of material that I recommend others go through, assuming they are interested in similar things as me.

Anyway, that's the vaporware, sorry for making it so long! I'm very excited to see how this Big Book of Study Stuffs evolves and what new projects it creates!

 

Yay vaporware! That's a really interesting goal, I for one would use an app like that all the time.

And what I meant was anything like anki or memrise that has more interaction; the doc is a guide but it asks no questions to the reader, it just shovels information down your throat. What I envisioned Cheap envisioning was a long term goal at having a personalized KM learning page that had sectioned topics with the grammar points laid out accompanied by related examples and questions. At least that seems like a reasonable goal.

Also, that's not a bad idea for getting the doc up, even if it was chopped into sections. And ya I agree that relating topics to buildings and floors could help, I'll try to work on that this weekend.

 

How does one achieve this:

 

Here's the research I've done so far, but all avenues need a better understanding of what we have available server-side: I see that www.kotobaminers.org is running Apache and PHP. @xemnes san, do you have advices about any of the following?

Google Apps Scripts is how you can automate your Google Docs. It has triggers for opening a document, but not saving or editing it (not yet; I filed a feature request), and for timer triggers that run as often as once a minute (but I don't think you want the webserver to get updates in the middle of an edit cycle), and it also allows for adding custom drop-down menus. I was able to add a new menu option "Email PDF to aldebrn" which worked nicely (gives confidence in Apps Scripts), and also possible is a "Publish PDF to Dropbox". Would it be possible to run a OAuth-secured endpoint in PHP so that a Google Apps Script could publish files to it?

Alternatively, you could look at the problem from just the server-side. The Google Drive API has a `watch` endpoint for push notifications that work through webhooks. The idea here is that you tell Google "whenever this document changes, send HTTP request to my server www.kotobaminers.org, which will then know what to do." That's the easy part, which that link spends some time detailing. But like with the Google Apps Script approach, we need something on the webserver which responds to the webhook, i.e., that recognizes the POST request from Google telling it the doc changed and grabbing the latest version: it could be something as simple as a single google-webhook.php file that does all this. Unless @xenmes san warns otherwise, I'll look into this a bit more.

(Netcraft tells me that the site is running WordPress? If that's the case, could HookPress be installed on it to respond to this? I can't quite tell if HookPress just translates WordPress-internal events to external webhooks, or if it can respond to external webhooks to update its own content.)

 

You lost me at Apache.

I think @Thasan is probably your best bet here. @xemnes did design the site, but Thasan takes care of all the back end stuff.

But yeah, @aldebrn you are crazy inspirational. Your knowledge of software is pretty amazing.

 

Our server side is not a problem, you can use this url for post:
http://redstone.kotobaminers.org/google-webhook.php
It dosent save anything than POST and GET data and information about uploaded files, but not file content
Any required response?

(I skipped most of those posts but if I got it right you want to save and publish pdf which is sent with POST?)

 

Close: the idea is to use Google Drive API's push notifications to inform kotobaminers.org that a Google Doc has been changed. The notification won't actually contain the file though. The POST message will encode the ID of the doc that's been updated: https://developers.google.com/drive/web/push#msg-format But you'll actively have to get the file yourself, through Google Drive API. It might be wise to keep a private list of docs that the server will update (currently just Prime-san's book here).

Yes, you'll want to respond with 200, otherwise Google will keep hitting you: https://developers.google.com/drive/web/push#responding

Before any of this will work, there's a process for registering with Google that you really do want it to send you push notifications: https://developers.google.com/drive/web/push#registering and the next couple of sections. Did you already complete this ?

After that you'll have to sign up for notifications for specific files: you may have to get the file ID from Prime-san or cheap-sensei, I'm not sure if non-authors can see it?

Also, the server needs to periodically repeat the "watch" request to Google to let Google know that push notifications are still desired: https://developers.google.com/drive/web/push#renew Not sure how you could best do this? cron job? You'll also want to be able to handle duplicate change notifications during the periods of overlap.

Even after re-reading their documentation, I'm not clear what it means for a document to have 'changed' --- will this notification be pushed for every character typed, or whenever it's autosaved, or after long periods of inactivity? I note the latest Google Docs doesn't even have a 'save' button . So maybe there'll be some experimentation to decide when to actually pull the PDF.

I can't see the source code for google-webhook.php obviously so I can't tell how far you've gotten in all this. I'm curious about how it all works and am tempted to set up the same functionality as a test, but for now I'll leave it in your hands Thasan-san let me know if I can help.

 

push notifications seems to require SSL and we dont have that

 

Ahh, it would have to be something that fundamental to stymie this yeah ok, assuming サンタさん isn't giving cheap-sensei an SSL certificate for クリスマス, we can go back to using Google Apps Script to poke that redstone.kotobaminers.org URL when one of the authors wants to "Publish" the document. I can look into how you use Google Drive API to get a PDF of the document, unless you want to do that.

The one thing I'm concerned about with this approach is the potential for spam, i.e., someone figures out what to POST to `google-webhook.php` and hammers on it, constantly causing the server to fetch a new PDF. This isn't the end of the world and I think it can be mostly ameliorated by the server rate-limiting its own requests to Google, plus with some security-through-obscurity, i.e., requiring the POST message to contain some "secret" password that you & Prime-san/cheap-sensei agree on offline (of course, without SSL/HTTPS, this is the tiniest obstacle to the scriptkiddies of the world, but hopefully will be enough to get started).

 

I'm not familiar with Google Drive API, so if you can do it, please do. That password is ok and I can make rate-limit too to.
Send PM if you want to start working with git

 

Can I just check my understanding of what is going on.

1. We update Prime's document
2. It gets exported to... (not sure where) This is a KM webpage though right?
3. This is indexed by google
4. Profit?

 

yes, thats right. but got small problem; our server has no SSL available (encrypted connection) which is required by google to send push notifications when document is edited.
When our server gets informatio that document is edited, it downloads and publish it to somewhere in our homepage so google can index it.

 

How do we get SSL then? (or is that something we can't get?)

 

65eur per year for single domain
335eur per year with wilcard
https://ssl.comodo.com/

 

€65 is not so bad for a year. I wonder how the google doc would look at a wiki page though.

 

I think its best to put separately somewhere, not to wikipage

 

but why not write it directly to wiki ?
Its now much easer to use when it has that visual editor

 

SSL is not at all an absolute requirement! SSL is needed to use the Google Drive push notifications where Google automatically tells KotobaMiners.org to get a new PDF (we're not sure how it automatically does this). The alternative is where in Google Docs, an author clicks a custom "Publish to Kotoba Miners" button and kotobaminers.org is thus manually notified to get an updated PDF; this manual route doesn't need SSL.

In both cases, the kotobaminers.org server fetches the latest PDF from Google.

I'll take a look at your wiki. There may be some RTF-to-MediaWiki converters out there... An alternative might be to have Google export HTML.

 

Oh I see what you mean: make prime-san rewrite his doc in the wiki well that would certainly solve a lot of problems!!!

 

If not directly to wiki, second best choise is pdf to somewhere in homepages, not to wiki or forums