# HG changeset patch # User Sverre Rabbelier # Date 1240063451 0 # Node ID f7497180d03724436c052e8baa5d03d27e6459db # Parent 1095b52ed6676940e5b828140418727ef7017d5c Add cron, the core of the job system Patch by: Sverre Rabbelier diff -r 1095b52ed667 -r f7497180d037 app/soc/cron/__init__.py --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/app/soc/cron/__init__.py Sat Apr 18 14:04:11 2009 +0000 @@ -0,0 +1,16 @@ +# +# Copyright 2009 the Melange authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""This module contains Melange cron jobs.""" diff -r 1095b52ed667 -r f7497180d037 app/soc/cron/job.py --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/app/soc/cron/job.py Sat Apr 18 14:04:11 2009 +0000 @@ -0,0 +1,143 @@ +#!/usr/bin/python2.5 +# +# Copyright 2009 the Melange authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Cron jobs. +""" + +__authors__ = [ + '"Sverre Rabbelier" ', + ] + + +import logging + +from google.appengine.ext import db +from google.appengine.runtime import DeadlineExceededError + +from soc.models.job import Job + + +class Handler(object): + """A handler that dispatches a cron job. + + The tasks that are mapped into tasks will be called when a worker + has claimed the job. However, there is no guarantee as to how long + the task will be allowed to run. If an Exception is raised the task + is automatically rescheduled for execution. + """ + + def __init__(self): + """Constructs a new Handler with all known jobs set. + """ + + self.tasks = {} + self.tasks['sendAcceptanceEmail'] = logging.info + + def claimJob(self, job_key): + """A transaction to claim a job. + """ + + job = Job.get_by_id(job_key) + + if job.status != 'waiting': + raise db.Rollback() + + job.status = 'started' + + if job.put(): + return job + else: + return None + + def freeJob(self, job_key): + """A transaction to free a job. + """ + + job = Job.get_by_id(job_key) + + job.status = 'waiting' + + return job.put() + + def failJob(self, job_key): + """A transaction to fail a job. + """ + + job = Job.get_by_id(job_key) + + job.errors += 1 + + if job.errors > 5: + job.status = 'aborted' + else: + job.status = 'waiting' + + job_id = job.key().id() + logging.warning("job %d now failed %d time(s)" % (job_id, job.errors)) + + return job.put() + + def finishJob(self, job_key): + """A transaction to finish a job. + """ + + job = Job.get_by_id(job_key) + job.status = 'finished' + + return job.put() + + def abortJob(self, job_key): + """A transaction to abort a job. + """ + + job = Job.get_by_id(job_key) + job.status = 'aborted' + + return job.put() + + def handle(self, job_key): + """Handle one job. + """ + + try: + job = db.run_in_transaction(self.claimJob, job_key) + + if not job: + # someone already claimed the job + return True + + if job.task_name not in self.tasks: + logging.error("Unknown job %s" % job.task_name) + db.run_in_transaction(self.abortJob, job_key) + return True + + task = self.tasks[job.task_name] + + # excecute the actual job + task(job) + + db.run_in_transaction(self.finishJob, job_key) + return True + except DeadlineExceededError, exception: + db.run_in_transaction(self.freeJob, job_key) + return False + except Exception, exception: + logging.exception(exception) + db.run_in_transaction(self.failJob, job_key) + return True + + +handler = Handler() diff -r 1095b52ed667 -r f7497180d037 app/soc/views/models/cron.py --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/app/soc/views/models/cron.py Sat Apr 18 14:04:11 2009 +0000 @@ -0,0 +1,126 @@ +#!/usr/bin/python2.5 +# +# Copyright 2009 the Melange authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Views for Cron. +""" + +__authors__ = [ + '"Sverre Rabbelier" ', + ] + + +from django import forms +from django import http + +from soc.logic import cleaning +from soc.logic import dicts +from soc.logic.models.priority_group import logic as priority_group_logic +from soc.logic.models.job import logic as job_logic +from soc.views.helper import access +from soc.views.helper import decorators +from soc.views.helper import dynaform +from soc.views.helper import widgets +from soc.views.models import base + +import soc.cron.job + + +class View(base.View): + """View methods for the Cron model. + """ + + def __init__(self, params=None): + """Defines the fields and methods required for the base View class + to provide the user with list, public, create, edit and delete views. + + Params: + params: a dict with params for this View + """ + + rights = access.Checker(params) + + new_params = {} + new_params['rights'] = rights + new_params['logic'] = priority_group_logic + + new_params['name'] = "Cron" + + new_params['django_patterns_defaults'] = [ + (r'^%(url_name)s/(?Padd)$', + 'soc.views.models.%(module_name)s.add', 'Add %(name_short)s'), + (r'^%(url_name)s/(?Ppoke)$', + 'soc.views.models.%(module_name)s.poke', 'Poke %(name_short)s'), + ] + + params = dicts.merge(params, new_params) + + super(View, self).__init__(params=params) + + def add(self, request, access_type, page_name): + group = priority_group_logic.getGroup(priority_group_logic.EMAIL) + + fields = { + 'priority_group': group, + 'task_name': 'sendAcceptanceEmail', + 'text_data': "O HI THAR", + } + + job_logic.updateOrCreateFromFields(fields) + + return http.HttpResponse("Done") + + def poke(self, request, access_type, page_name): + """ + """ + + order = ['-priority'] + query = priority_group_logic.getQueryForFields(order=order) + groups = priority_group_logic.getAll(query) + handler = soc.cron.job.handler + + groups_completed = 0 + jobs_completed = 0 + + for group in groups: + filter = { + 'priority_group': group, + 'status': 'waiting', + } + + query = job_logic.getQueryForFields(filter=filter) + jobs = job_logic.getAll(query) + + for job in jobs: + job_key = job.key().id() + good = handler.handle(job_key) + + if not good: + break + + jobs_completed += 1 + + groups_completed += 1 + + response = 'Completed %d jobs and %d groups completed.' % ( + jobs_completed, groups_completed) + + return http.HttpResponse(response) + + +view = View() + +add = view.add +poke = view.poke diff -r 1095b52ed667 -r f7497180d037 app/soc/views/sitemap/build.py --- a/app/soc/views/sitemap/build.py Sat Apr 18 14:03:49 2009 +0000 +++ b/app/soc/views/sitemap/build.py Sat Apr 18 14:04:11 2009 +0000 @@ -28,6 +28,7 @@ #from soc.views.models import club_app #from soc.views.models import club_admin #from soc.views.models import club_member +from soc.views.models import cron from soc.views.models import document from soc.views.models import host from soc.views.models import job @@ -83,6 +84,7 @@ #sitemap.addPages(club_admin.view.getDjangoURLPatterns()) #sitemap.addPages(club_app.view.getDjangoURLPatterns()) #sitemap.addPages(club_member.view.getDjangoURLPatterns()) +sitemap.addPages(cron.view.getDjangoURLPatterns()) sitemap.addPages(document.view.getDjangoURLPatterns()) sitemap.addPages(host.view.getDjangoURLPatterns()) sitemap.addPages(job.view.getDjangoURLPatterns())