„FederationStats” változatai közötti eltérés

Innen: KIFÜ Wiki
(db:setup)
(Creating IdPs)
(7 közbenső módosítás, amit 2 másik szerkesztő végzett, nincs mutatva)
4. sor: 4. sor:
 
Federation visualization project
 
Federation visualization project
 
* source (ruby on rails) https://repo.niif.hu/gitweb/gitweb.cgi?p=federation-stats.git;a=summary
 
* source (ruby on rails) https://repo.niif.hu/gitweb/gitweb.cgi?p=federation-stats.git;a=summary
* live demo https://papigw.aai.niif.hu/fedstats
+
* live demo http://eduid.hu/stats
  
 
== Running the sample project ==
 
== Running the sample project ==
22. sor: 22. sor:
  
 
  ENTITYID #ENTITYID#
 
  ENTITYID #ENTITYID#
 +
APIKEY #API_KEY#
 
  DATE yyyy-mm-dd
 
  DATE yyyy-mm-dd
 
   
 
   
37. sor: 38. sor:
  
 
  ENTITYID https://idp.niif.hu/idp/shibboleth
 
  ENTITYID https://idp.niif.hu/idp/shibboleth
 +
APIKEY 0123.......
 
  DATE 2009-03-18
 
  DATE 2009-03-18
 
   
 
   
60. sor: 62. sor:
 
  2        | https://idp.sch.bme.hu:443/opensso/sp/test
 
  2        | https://idp.sch.bme.hu:443/opensso/sp/test
  
 +
== Running the log statistics collector ==
 +
This following script can be used the collect statistics from the idp audit logs of Shibboleth 2 IdP generated on the day before running. It is based on Peter Schober's audit_r7.py, and good for run from daily cronjob:
 +
 +
<source lang="bash">
 +
#!/bin/bash
 +
 +
#Config section
 +
PARSER_COMMAND="/opt/shibboleth-idp/bin/audit_r7.py"
 +
SOURCEDIR="/opt/shibboleth-idp/logs"
 +
TARGETDIR="/tmp"
 +
 +
ENTITYID="idp-entity-id"
 +
APIKEY="aaa..."
 +
LOCATION2PUT="https://fedstats.example.org/import_stats"
 +
 +
DATE=`date -d "yesterday" +"%Y-%m-%d"`
 +
SOURCEFILE="$SOURCEDIR/idp-audit-$DATE.log"
 +
 +
#Should not edit below this
 +
 +
if [ -f $SOURCEFILE ]
 +
then
 +
    LOGINS=`$PARSER_COMMAND -l $SOURCEFILE`
 +
    UNIQUE_LOGINS=`$PARSER_COMMAND -u $SOURCEFILE`
 +
    SERVICES=`$PARSER_COMMAND -p $SOURCEFILE | sed '/^[0-9]/p' -n`
 +
 +
    TARGETFILE="stat-$DATE.log"
 +
 +
echo "ENTITYID $ENTITYID
 +
APIKEY $APIKEY
 +
DATE $DATE
  
 +
STAT AUTH
 +
$LOGINS
  
== Running the log statistics collector ==
+
STAT USER_COUNT
This following script can be used the collect statistics from the idp audit logs of Shibboleth 2 IdP. It is based on Peter Scober's audit_r7.py:
+
$UNIQUE_LOGINS
 +
 
 +
STAT SSO_TO_SERVICE
 +
$SERVICES
 +
" > $TARGETDIR/$TARGETFILE
 +
 
 +
    wget -q --no-check-certificate --post-file=$TARGETDIR/$TARGETFILE $LOCATION2PUT -O /dev/null
 +
    rm $TARGETDIR/$TARGETFILE
 +
fi
 +
 
 +
</source>
 +
 
 +
The script below can be used the collect statistics from all the idp audit logs placed in a folder.
  
 
<source lang="bash">
 
<source lang="bash">
 
#!/bin/bash
 
#!/bin/bash
  
FILES="log-directory/idp-audit-*.log"
+
#Config section
TARGETDIR="/tmp/stats"
+
PARSER_COMMAND="/opt/shibboleth-idp/bin/audit_r7.py"
 +
SOURCEDIR="/opt/shibboleth-idp/logs"
 +
TARGETDIR="/tmp"
 +
 
 
ENTITYID="idp-entity-id"
 
ENTITYID="idp-entity-id"
AUDIT="/path/to/audit_r7.py"
+
APIKEY="aaa..."
 +
LOCATION2PUT="https://fedstats.example.org/import_stats"
 +
 
 +
FILES="idp-audit-*.log"
  
 +
#Should not edit below this
 +
cd $SOURCEDIR
 
for f in $FILES
 
for f in $FILES
 
do
 
do
        echo "Processing $f file..."
+
  if [ -f $f ]
        DATE=${f:10:10}
+
  then
        LOGINS=`$AUDIT -l $f`
+
    echo "Processing $f file..."
        USERS=`$AUDIT -u $f`
+
    DATE=${f:10:10}
        SERVICES=`$AUDIT -p $f | sed '/^[0-9]/p' -n`
+
    LOGINS=`$PARSER_COMMAND -l $f`
        FILENAME="stat-$f"
+
    UNIQUE_LOGINS=`$PARSER_COMMAND -u $f`
 +
    SERVICES=`$PARSER_COMMAND -p $f | sed '/^[0-9]/p' -n`
 +
 
 +
    TARGETFILE="stat-$DATE.log"
  
        echo "ENTITYID $ENTITYID
+
    echo "ENTITYID $ENTITYID
 +
APIKEY $APIKEY
 
DATE $DATE
 
DATE $DATE
  
89. sor: 148. sor:
  
 
STAT USER_COUNT
 
STAT USER_COUNT
$USERS
+
$UNIQUE_LOGINS
  
 
STAT SSO_TO_SERVICE
 
STAT SSO_TO_SERVICE
 
$SERVICES
 
$SERVICES
" > $TARGETDIR/$FILENAME
+
" > $TARGETDIR/$TARGETFILE
  
done #for files
+
    wget -q --no-check-certificate --post-file=$TARGETDIR/$TARGETFILE $LOCATION2PUT -O /dev/null
 +
    rm $TARGETDIR/$TARGETFILE
 +
  fi
 +
done
 
</source>
 
</source>
  
101. sor: 163. sor:
 
The federation statistics rails project contains the <code>script/stat_parser/file.rb</code> command which can process the statistics format and load the data to the database. Note that this script currently contains an absolute path for the <code>script/runner</code> script, so you must fix this before use.
 
The federation statistics rails project contains the <code>script/stat_parser/file.rb</code> command which can process the statistics format and load the data to the database. Note that this script currently contains an absolute path for the <code>script/runner</code> script, so you must fix this before use.
  
 +
== Using HTTP-Post to feed the database ==
 +
When deployed, the rails project provides a <code>/import_stats</code> URL to which one could POST the generated statistics file.
 +
 +
== Creating IdPs ==
 +
Use the rails console to create new idps:
 +
 +
$ RAILS_ENV=production script/console
 +
 +
>> Entity.create :name => 'foo', :entity_type => 'idp'
 +
 +
=> #<Entity id: 1, name: "foo", entity_type: "idp", created_at: "2010-11-29 14:55:40", updated_at: "2010-11-29 14:55:40", api_key: "da9l233a45698fa5c4a252e301e3da2sf5ece24e">
  
 
[[Category:Csonkok]]
 
[[Category:Csonkok]]

A lap 2010. november 30., 15:20-kori változata

Federation usage statistics


Federation visualization project

Running the sample project

  • Install Ruby
  • Install Rails (gem install rails)
  • Setup a development.sqlite3 database with the rake db:setup command
  • Fire up script/server, it will run the project on localhost:3000

Statistic types

Currently we have the following types of statistics:

  • Unique users per day (USER_COUNT)
  • AuthnResponse per day (AUTH)
  • AuthnResponse per service per day (SSO_TO_SERVICE)

Log statistics format

The following simple format is used to convey statistics from IdPs to the central module - the white spaces (new lines) are important:

ENTITYID #ENTITYID#
APIKEY #API_KEY#
DATE yyyy-mm-dd

STAT #STAT_ID#
xxxx

STAT #STAT_ID#
yyyy

STAT #STAT_ID#
ww | #PEER_ENTITY_1#
zz | #PEER_ENTITY_2#

The following sample might help understanding the format:

ENTITYID https://idp.niif.hu/idp/shibboleth
APIKEY 0123.......
DATE 2009-03-18

STAT AUTH
68 logins

STAT USER_COUNT
16 unique userids

STAT SSO_TO_SERVICE
1        | urn:geant:niif.hu:niifi:sp:register.ca.niif.hu
12       | https://repo.niif.hu/shibboleth
1        | https://sandbox.aai.niif.hu/shibboleth
5        | https://sysmonitor.hbone.hu/shibboleth
10       | https://www.ki.iif.hu/shibboleth
1        | https://noc6.vh.hbone.hu/shibboleth
21       | https://webadmin.iif.hu/shibboleth
3        | https://rrd-ma.perfsonar.vh.hbone.hu/shibboleth
7        | https://ugyeletes.vh.hbone.hu/shibboleth
2        | https://noc.grid.niif.hu/shibboleth
1        | https://wiki.voip.niif.hu/shibboleth
2        | https://netmonitor.hbone.hu/shibboleth
2        | https://idp.sch.bme.hu:443/opensso/sp/test

Running the log statistics collector

This following script can be used the collect statistics from the idp audit logs of Shibboleth 2 IdP generated on the day before running. It is based on Peter Schober's audit_r7.py, and good for run from daily cronjob:

#!/bin/bash

#Config section
PARSER_COMMAND="/opt/shibboleth-idp/bin/audit_r7.py"
SOURCEDIR="/opt/shibboleth-idp/logs"
TARGETDIR="/tmp"

ENTITYID="idp-entity-id"
APIKEY="aaa..."
LOCATION2PUT="https://fedstats.example.org/import_stats"

DATE=`date -d "yesterday" +"%Y-%m-%d"`
SOURCEFILE="$SOURCEDIR/idp-audit-$DATE.log"

#Should not edit below this

if [ -f $SOURCEFILE ]
then
    LOGINS=`$PARSER_COMMAND -l $SOURCEFILE`
    UNIQUE_LOGINS=`$PARSER_COMMAND -u $SOURCEFILE`
    SERVICES=`$PARSER_COMMAND -p $SOURCEFILE | sed '/^[0-9]/p' -n`

    TARGETFILE="stat-$DATE.log"

echo "ENTITYID $ENTITYID
APIKEY $APIKEY
DATE $DATE

STAT AUTH
$LOGINS

STAT USER_COUNT
$UNIQUE_LOGINS

STAT SSO_TO_SERVICE
$SERVICES
" > $TARGETDIR/$TARGETFILE

    wget -q --no-check-certificate --post-file=$TARGETDIR/$TARGETFILE $LOCATION2PUT -O /dev/null
    rm $TARGETDIR/$TARGETFILE
fi

The script below can be used the collect statistics from all the idp audit logs placed in a folder.

#!/bin/bash

#Config section
PARSER_COMMAND="/opt/shibboleth-idp/bin/audit_r7.py"
SOURCEDIR="/opt/shibboleth-idp/logs"
TARGETDIR="/tmp"

ENTITYID="idp-entity-id"
APIKEY="aaa..."
LOCATION2PUT="https://fedstats.example.org/import_stats"

FILES="idp-audit-*.log"

#Should not edit below this
cd $SOURCEDIR
for f in $FILES
do
  if [ -f $f ]
  then
    echo "Processing $f file..."
    DATE=${f:10:10}
    LOGINS=`$PARSER_COMMAND -l $f`
    UNIQUE_LOGINS=`$PARSER_COMMAND -u $f`
    SERVICES=`$PARSER_COMMAND -p $f | sed '/^[0-9]/p' -n`

    TARGETFILE="stat-$DATE.log"

    echo "ENTITYID $ENTITYID
APIKEY $APIKEY
DATE $DATE

STAT AUTH
$LOGINS

STAT USER_COUNT
$UNIQUE_LOGINS

STAT SSO_TO_SERVICE
$SERVICES
" > $TARGETDIR/$TARGETFILE

    wget -q --no-check-certificate --post-file=$TARGETDIR/$TARGETFILE $LOCATION2PUT -O /dev/null
    rm $TARGETDIR/$TARGETFILE
  fi
done

Feeding the database with the statistics

The federation statistics rails project contains the script/stat_parser/file.rb command which can process the statistics format and load the data to the database. Note that this script currently contains an absolute path for the script/runner script, so you must fix this before use.

Using HTTP-Post to feed the database

When deployed, the rails project provides a /import_stats URL to which one could POST the generated statistics file.

Creating IdPs

Use the rails console to create new idps:

$ RAILS_ENV=production script/console

>> Entity.create :name => 'foo', :entity_type => 'idp'

=> #<Entity id: 1, name: "foo", entity_type: "idp", created_at: "2010-11-29 14:55:40", updated_at: "2010-11-29 14:55:40", api_key: "da9l233a45698fa5c4a252e301e3da2sf5ece24e">