Difference between revisions of "Stats Design"

From Dreamwidth Notes
Jump to: navigation, search
(Dump some stuff in)
 
(18 intermediate revisions by 3 users not shown)
Line 1: Line 1:
−
This is a placeholder for the Stats design page.
+
{{Update}}
  
−
http://bugs.dwscoalition.org/show_bug.cgi?id=124
+
This is a a loosely organized collection of notes and links for the Stats feature design. (Note: this is for the new stats system only - there's an existing legacy system too.)
−
Table definition:
+
 
−
CREATE TABLE statkeylist (  
+
[[Project_Teams#Site_statistics|Stats project team]]
−
statkeyid int unsigned NOT NULL auto_increment,  
+
 
−
name varchar(255) default NULL,  
+
== Table definitions ==
 +
 
 +
<source lang="sql">CREATE TABLE statkeylist (
 +
    statkeyid int unsigned NOT NULL auto_increment,  
 +
    name varchar(255) default NULL,  
 
   
 
   
−
PRIMARY KEY (statkeyid),  
+
    PRIMARY KEY (statkeyid),  
−
UNIQUE KEY (name)  
+
    UNIQUE KEY (name)  
−
)  
+
)
  
 
CREATE TABLE site_stats (  
 
CREATE TABLE site_stats (  
−
category_id INT UNSIGNED NOT NULL,  
+
    category_id INT UNSIGNED NOT NULL,  
−
key_id INT UNSIGNED NOT NULL,  
+
    key_id INT UNSIGNED NOT NULL,  
−
insert_time INT UNSIGNED NOT NULL,  
+
    insert_time INT UNSIGNED NOT NULL,  
−
value INT UNSIGNED NOT NULL,  
+
    value INT UNSIGNED NOT NULL,  
 
   
 
   
−
-- FIXME: This is good for retrieving data for a single category+key, but  
+
    -- FIXME: This is good for retrieving data for a single category+key, but  
−
-- maybe not as good if we want all keys for the category, with a limit on  
+
    -- maybe not as good if we want all keys for the category, with a limit on  
−
-- time (ie, last 5 entries, or last 2 weeks). Do we need an extra index?  
+
    -- time (ie, last 5 entries, or last 2 weeks). Do we need an extra index?  
−
INDEX (category_id, key_id, insert_time)  
+
    INDEX (category_id, key_id, insert_time)  
−
)  
+
)</source>
 +
 
 +
== Storing and retrieving statistics ==
 +
 
 +
[https://github.com/dreamwidth/dw-free/blob/develop/cgi-bin/DW/StatStore.pm DW::StatStore]
 +
 
 +
== Statistics items ==
 +
 
 +
* [https://github.com/dreamwidth/dw-free/blob/develop/cgi-bin/DW/StatData.pm DW::StatData]
 +
* [https://github.com/dreamwidth/dw-free/blob/develop/cgi-bin/DW/StatData/ DW::StatData::*]
 +
 
 +
== Collecting statistics ==
 +
 
 +
This works by massaging, aggregating, and or compiling (as appropriate) data from various places and storing the results using DW::StatStore.
 +
 
 +
1. [https://github.com/dreamwidth/dw-free/blob/develop/etc/stats-collection.conf etc/stats-collection.conf] should contain something like:
 +
<source lang="text">## collect and store all stats under the accounts category
 +
## hangs if you don't enclose the asterisk in quotes
 +
# accounts: "*"
 +
## collect and store only these stats under the accounts category
 +
# accounts: [ identity, personal, redirect ]
 +
accounts: "*"
 +
active: "*"
 +
paid: "*"
 +
</source>
 +
 
 +
2. [https://github.com/dreamwidth/dw-free/blob/develop/bin/worker/stats-collection bin/worker/stats-collection] is the utility that handles stats collection. Run at least once per 24 hours, either from your crontab or by listing it in [https://github.com/dreamwidth/dw-free/blob/develop/etc/workers.conf etc/workers.conf].
 +
 
 +
3. (only needed if you chose to use [https://github.com/dreamwidth/dw-free/blob/develop/etc/workers.conf etc/workers.conf]): add a line to the "all" section reading:
 +
<source lang="text">  stats-collection: 1
 +
</source>(note: this is for development only, as it assumes you only have one server - in production environments, you will usually list it in the section for a specific server)
 +
 
 +
== Displaying statistics ==
 +
 
 +
* [https://github.com/dreamwidth/dw-free/blob/develop/views/stats/site.tt views/stats/site.tt] (public stats)
 +
* [https://github.com/dreamwidth/dw-free/blob/develop/views/admin/stats.tt views/admin/stats.tt] (restricted stats)
 +
* [https://github.com/dreamwidth/dw-free/blob/develop/cgi-bin/DW/Controller/SiteStats.pm DW::Controller::SiteStats.pm] (the stats page controller for both public and restricted stats)
 +
* [https://github.com/dreamwidth/dw-free/blob/develop/cgi-bin/DW/Controller/Graphs.pm DW::Controller::Graphs.pm] (the stats graphs controller for both public and restricted stats)
  
−
Storing statistics: http://hg.dwscoalition.org/dw-free/file/67e1042df080/cgi-bin/DW/StatStore.pm
+
[[Category: Development]]
−
Statistics: http://hg.dwscoalition.org/dw-free/file/67e1042df080/cgi-bin/DW/StatData.pm and http://hg.dwscoalition.org/dw-free/file/67e1042df080/cgi-bin/DW/StatData/
+
−
Collecting statistics:
+
−
Displaying statistics:
+

Latest revision as of 10:08, 4 August 2014

Needs Update: This page has been found in need of an update. Information may be wrong or outdated!

This is a a loosely organized collection of notes and links for the Stats feature design. (Note: this is for the new stats system only - there's an existing legacy system too.)

Stats project team

Table definitions

CREATE TABLE statkeylist (
    statkeyid INT UNSIGNED NOT NULL AUTO_INCREMENT, 
    name VARCHAR(255) DEFAULT NULL, 
 
    PRIMARY KEY (statkeyid), 
    UNIQUE KEY (name) 
)
 
CREATE TABLE site_stats ( 
    category_id INT UNSIGNED NOT NULL, 
    key_id INT UNSIGNED NOT NULL, 
    insert_time INT UNSIGNED NOT NULL, 
    VALUE INT UNSIGNED NOT NULL, 
 
    -- FIXME: This is good for retrieving data for a single category+key, but 
    -- maybe not as good if we want all keys for the category, with a limit on 
    -- time (ie, last 5 entries, or last 2 weeks). Do we need an extra index? 
    INDEX (category_id, key_id, insert_time) 
)

Storing and retrieving statistics

DW::StatStore

Statistics items

Collecting statistics

This works by massaging, aggregating, and or compiling (as appropriate) data from various places and storing the results using DW::StatStore.

1. etc/stats-collection.conf should contain something like: 

## collect and store all stats under the accounts category
## hangs if you don't enclose the asterisk in quotes
# accounts: "*"
## collect and store only these stats under the accounts category
# accounts: [ identity, personal, redirect ]
accounts: "*"
active: "*"
paid: "*"

2. bin/worker/stats-collection is the utility that handles stats collection. Run at least once per 24 hours, either from your crontab or by listing it in etc/workers.conf.

3. (only needed if you chose to use etc/workers.conf): add a line to the "all" section reading: 

  stats-collection: 1
(note: this is for development only, as it assumes you only have one server - in production environments, you will usually list it in the section for a specific server)

Displaying statistics

Retrieved from "http://wiki.dreamwidth.net/wiki/index.php?title=Stats_Design&oldid=12072"