Archive for category Plone

Export Plone 3 folder to zip file

Motivation

To add functionality to a Plone 3 site to enable a user to download the contents of a folder and all subfolders as a .zip file.

Environment in which this tool was developed

Plone 3.3.1
Zope 2.10.9-final
Python 2.4.6
linux2

Important Notes

Excluded Content

The following types of content are excluded from the .zip file:

  • Links
  • Events
  • Empty Folders

Permissions and workflow state

The .zip file created by this tool includes only files and folders.  Permissions and workflow states on folders or content items are not retained in any way.  Permissions, however, are not bypassed — i.e., only content objects on which the current user has the View permission are included.

HTML Documents: Pages and News Items

“Pages” (a.k.a. “Documents”) and “News Items” are processed in the following way:

  • If the .html file extension is missing, it is added
  • A complete, yet simple, HTML document is created for the content — i.e., the Plone “wrapper” is removed.
  • The body of the document consists of the “cooked” document content, with the addition of an H1 element at the top containing the document title.
    If the document has a description, it is inserted in a paragraph element below the title.
  • The document creator and last modified date are added below the document content.

The Pieces

External module

The easiest way to implement is to create an “old-style” Zope product, which is simply a Python package, and put it in the products directory.  In my case, that directory is /opt/plone/zeoserver/products.  You may want to put an empty file called refresh.txt in the root of the package — this can help in development to avoid having to restart Zope clients to pick up changes — although you do still have to re-save the External Method that references the module, and I found that restarting the clients was often required anyway.  In the Python package, create a subdirectory, not a subpackage, called Extensions, and create your code module there.

For purposes of this article, I’ll call the product package is ExportTool, and the module export.py.

Note: To use the code as-is, you’ll need to create /opt/plone/temp and make it writeable by the user Unix under which the Plone clients run.  Alternatively, you could assign the TEMPDIR global variable in the module to tempfile.gettempdir().

Source:

import cgi
import os
import shutil
import subprocess
import tempfile

TEMPDIR = '/opt/plone/temp'

DOC_TEMPLATE = """\
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
    <title>%(title)s</title>
  </head>
  <body>
    <h1>%(title)s</h1>
    %(body)s
    <hr/>
    <p>
      Created by: %(creator)s<br/>
      Last modified: %(modified)s
    </p>
  </body>
</html>
"""

def export_folder(context, response):

    transform_tool = context.portal_transforms
    # temp dir for this export job
    tempdir = tempfile.mkdtemp(dir=TEMPDIR)

    def _export(folder, tempdir):
        # create dir into which folder contents will be exported as files
        folder_path = folder.getPhysicalPath()[1:]
        export_dir = os.path.join(tempdir, os.path.join(*folder_path))
        os.makedirs(export_dir)
        for obj in folder.getFolderContents(full_objects=True):
            if obj.portal_type == 'Folder':
                # recursive call
                _export(obj, tempdir)
            elif obj.portal_type in ['Image', 'File', 'Document', 'News Item']:
                filename = obj.getId()
                if obj.portal_type in ['Image', 'File']:
                    content = obj.data
                else:
                    if not filename.endswith('.html'):
                        filename += '.html'
                    body = obj.CookedBody()
                    description = obj.Description()
                    if description:
                        body = '<p>%s</p>\n%s' % (cgi.escape(description, quote=True), body)
                    content = DOC_TEMPLATE % {
                        'body': body,
                        'title': cgi.escape(obj.Title(), quote=True),                        
                        'modified': context.toLocalizedTime(obj.ModificationDate(), long_format=1),
                        'creator': obj.Creator(),
                        }
                outfile = open(os.path.join(export_dir, filename), 'wb')
                outfile.write(content)
                outfile.close()

    try:
        # export the content
        _export(context, tempdir)
        # create a zip file            
        os.chdir(tempdir)
        zipprefix = '-'.join(context.getPhysicalPath()[1:])
        subprocess.call(['/usr/bin/zip', '-r', zipprefix, 'intranet'])
        zipname = zipprefix + '.zip'
        zipped = os.path.join(tempdir, zipname)
        # set response headers
        response.setHeader('Content-Type', 'application/zip')
        response.setHeader('Content-Disposition', 'attachment; filename=%s' % zipname)
        # write zip file to response
        z = open(zipped, 'rb')
        while 1:
            chunk = z.read(8192)
            if chunk:
                response.write(chunk)
            else:
                break
        z.close()        
    finally:
        # delete the temp dir
        shutil.rmtree(tempdir)        
    return

Zope External Method

id: export_folder_to_zip
Module Name: ExportTool.export
Function Name: export_folder

Zope Script (Python)

id: exportFolderToZip

Source:

if context.portal_type == 'Folder' and len(context.getPhysicalPath()) > 3:
    container.export_folder_to_zip(context, container.REQUEST.RESPONSE)

Note: The guard limiting path length is to prevent downloading top-level folders (path element 1 is a slash, 2 is the Plone site, 3 is the first folder).

Add Action to Folder portal type

Title: Zip
Id: zip
URL (Expression): string:${folder_url}/exportFodlerToZip
Condition (Expression): (blank)
Permission: Modify portal content
Category: Object
Visible? (check)

Advertisements

Leave a comment

GenericSetup and separation of concerns

As I continue my investigation of and attempt to implement a GenericSetup approach to customizations for my Plone site, I am seeing more clearly the need for this transition, even if the reasoning behind some of the design choices escapes me.  

In TTW customization of Zope 2 there was no natural way to cleanly separate customizations from default functionality.  Perhaps more significantly, the system did not really provide a canonical way of separating presentation from business logic.  Zope Page Templates (and the older DTML methods) were, and are, too powerful for their own good.  On the other side “Script (Python)” scripts are too limited in power (due to security restrictions) and scope to be able to handle general purpose business logic.  This state of affairs leads almost unavoidably to “spaghetti code” strung across a mixture of page templates, DTML methods, “Script (Python)” scripts and “External Methods”. 

The addition of “Controller Page Template”, “Controller Python Script”, and “Controller Validator” objects appears to have been an earlier attempt to address the problem of a lack of a clear MVC-style “controller”.  While these objects provided some conveniences, at least for form processing, they didn’t eliminate the deeper problem.  I’m still not sure that Plone 3/Zope 2/Five has really resolved the fundamental issue, but definite progress has been made.  

I do regret the profusion in GenericSetup of XML configuration files (a la Java web apps) which IMO humans should not have to read, much less write. The use of XML-like “ZCML” for the main configuration files seems particularly strange. There must have been some initial reason for not using XML, but now it just seems arbitrary and bewildering.  Personally, I would much prefer a pure Python implementation, but what do I know?

, , , , , ,

Leave a comment

Plone: Transitioning from TTW customization to GenericSetup

So, I buy into the idea of GenericSetup, but I’m finding the transition to using it for my site customizations rather painful.  In part, this is due to the fact that most of the relevant documentation on plone.org is a bit old and thin.  It seems that the best bet for a quick start is to use the generator script that ships with DIYPloneStyle.  There is still a certain amount of guessing or trial and error, but at least the configuration file comments are helpful.  You can also find examples in the CMFPlone package.

My Plone site goes back to Plone 2.0.5, so I went though the painful transition to Archetypes-based content types. That change was mandatory to get to Plone 2.1+, and now I’m seeing the writing on the wall in Plone 3 with GenericSetup and Zope 3 technologies.  While I worried in a previous post that this change was adding complexity that would make customizing Plone more difficult for the casual developer, I also recognize that the new approach provides a level of modularity across the site that was not possible with TTW customization alone.  But since I was fairly heavily invested in the TTW approach, I am now having to untangle all the little things that were added or changed in various places, from site properties to actions and workflow.

, , ,

Leave a comment

My life with Plone, part 4

Continues parts 1, 2, and 3.

Add-on products

In the Zope world a “product” is more or less equivalent to an “add-on” or a “plugin” in comparable frameworks — the encapsulation of some functionality that extends the core system by “installing” it though a standard procedure.  The CMF is actually a bundle of products that extend Zope, and Plone is largely built on the CMF. Over time, more and more products originally built on the CMF (placeful workflow) or Zope 2 (external editor) have been bundled with Plone, and some of those have integrated with Plone’s management UI.  This is generally good; in fact, for future compatibility reasons, I tend to prefer sticking as closely as possible to Plone’s distribution product bundle and minimizing the number of additional products added to the system. This of course limits the functionality I can offer to users, but also limits the risks. If you install a product that is not shipped with Plone you should be prepared to support it yourself.

Performance and caching

I find this excerpt from the article Plone patterns and best practices on plone.org says it well:

Here’s a little fact we should admit up front: Plone is slow. Not as slow as it used to be, not as slow that it’s useless, but neither Zope nor Plone have ever been sold on speed. In general, Plone is a content production system, not a content delivery system, and has been written for features and ease-of-use — not for blazing speed. (We use caching to improve speed, or separate delivery platforms altogether.)

[Author’s emphasis; accessed 23 December 2008.–DCS]

Performance can be a significant, if solvable, problem with Plone and there are many strategies for dealing with the issue. Finding the best solution, however, for the your particular system requirements and combination of time and skills can be a challenge.  Fortunately, changes since Plone 3.0, such as greater reliance on catalog data, have improved out-of-the-box performance. OTOH Plone makes very heavy use of CSS and JavaScript, and I’ve found that one of the biggest performance boosts comes from front-end caching of this content. I use mod_cache in Apache 2.2 for this and it works well. Zope 2 has long had integrated caching capabilities, but the machanisms for applying them (via the ZMI) are rather crude.  CacheFu promises an integrated Plone solution, but I encountered a roadblock in version 1.1.2 that prevented me from implementing it in my system.

If you have a zoecluster installation with a front-end proxy server, then you can also implement simple load balancing. I use three zeocluster clients, two for user access and one for administrative tasks such as updating the catalog. With Apache and mod_proxy_balancer I can distribute requests between the first two clients:

<Proxy balancer://zeocluster>
    BalancerMember http://localhost:8080
    BalancerMember http://localhost:8081
    ...
</Proxy>

RewriteRule ^/(.*) balancer://zeocluster/VirtualHostBase/https/my.host:443/plone/VirtualHostRoot/$1 [L,P]

I should mention that PTProfiler is indispensable for diagnosing performance issues in Zope page templates. More than once I have been able to isolate a performance problem in my own code using this tool.

The community

The Plone community is large and very international. On the whole I would say my experience has been about average for an open source project — that is, sometimes there’s good help, sometimes not.  Because of the complexity of a Plone installation — including Python, Zope, Plone, third-party products, customizations, etc., not to mention external systems such as Apache, LDAP, etc. — troubleshooting problems can be tricky and knowing to whom a problem is best addressed isn’t always clear.  Again, this is why I prefer to stick close to the distribution: If I have a problem it’s more likely not to be my fault, and maybe there’s a better chance I can get help.

, , , ,

2 Comments

My life with Plone, part 3

A continuation of parts 1 and 2.

Workflow

Plone’s workflow functionality, which is the portal_workflow tool of the underlying Zope CMF, can do just about anything you want it to do, provided that you can figure out how to do it. Plone itself still doesn’t have a UI for creating and configuring workflow, so you have to do it in the ZMI, which AFAICT has changed little in the time I have used Zope and the CMF.  Plone has fairly recently integrated placeful workflow, adding the ability to override sitewide workflows at the folder level, which is an important consideration in a large and complex site.  Hopefully, they’ve fixed the bugs in the UI (both Plone and the ZMI) which I encountered when I first tried using it.

There are a couple things about CMF/Plone workflow that are not exactly intuitive, or at least are different from how other CMS systems implement workflow functionality:

  1. A content item is always under workflow, if its content type is configured to use workflow at all. In other words, a content item doesn’t go “through” workflow; rather, the item is bound to its workflow.
  2. Workflow has no security per se.  In other words, access to objects in various workflow states is determined by the security settings on the objects in the ZODB. Workflow transitions implement changes in ZODB security settings.

An access control gotcha

Zope security settings are, so to speak, atomic.  Inheritance of permissions can be toggled on/off at any level of the object hierarchy and explicitly overridden on any object.  So far, so good.  However, unlike file systems, access to objects is not strictly dependent on access to their containers.  For example, a “private” folder which is not publicly accessible can contain “published” or “public draft” items which are.  The security problem is that content authors may assume, not without reason, that any content they put in a private folder will also be private.  This is a deep issue because CMF workflow explicitly sets object permissions and Zope security only considers the permissions on an object, not its container (factoring in inherited permissions, of course).  A workaround I have implemented is a modification of the default workflow to force setting the initial state of an object to “private” if its parent object is in the private state.  At least this provides a modicum of protection to unwitting content authors; they have to explicity “make visible” or “publish” these items to make them more widely accessible.

1. Wrote a script hideInPrivateParent:

## Script (Python) "hideInPrivateParent"
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=state_change
##title=Transition state_change.object to private state if parent object is private
##
wftool = context.portal_workflow
PRIVATE_TRANSITION = 'hide'
PRIVATE_STATE = 'private'
obj = state_change.object
parent = obj.aq_inner.getParentNode()
if parent is not None:
    try:
        if wftool.getInfoFor(parent, 'review_state') == PRIVATE_STATE:
            wftool.doActionFor(obj, PRIVATE_TRANSITION)
    except:
        pass

2. Created a new workflow state called “initialized” and set it as the default workflow state.

3. Created a new workflow transition call “initialize”.  Set the “trigger type” to “automatic”.  Set the “destination state” to “visible”. Set “script (after)” to hideInPrivate Parent.

The result is that when a new content item is created with this workflow, its state will be private if its parent object (typically a folder) is private, otherwise it will be “visible”.

Continued in Part 4.

, ,

1 Comment

My life with Plone, part 2

In part 1 I discussed documentation, support for previous releases, upgrading, backup/restore and versioning.

The object database

As I said in Part 1, the ZODB is very solid. I find, however, that the tools for analyzing the contents of the database — at least those that ship with Plone’s Zope/CMF — are relatively weak. It feels like the system needs a GUI app for administrators. As it is, you either deal with the limited functionality of the ZMI, or you use plain old Python. I miss the power of relational databases to generate ad hoc reports with SQL statements, or even something like Django’s ORM methods. Part of the problem I suppose is poor documentation of the Zope API, and it can take some detective work and trial and error (even if you do use DocFinderTab) to find not only the methods you need, but the correct way to call them. This in turn is to some extent the fault of the Python language itself in that function parameters and return values are not typed as in Java, for example. There are certainly advantages to Python’s approach — and everything considered Python is still my preferred programming language — but I do think it often results in weaker documentation. Even the official documentation of the Python language is, to me, often maddeningly vague with respect to input and output of standard library functions.

Managing permissions, users and groups

Zope has a complex, flexible, and extremely granular security model, almost too much so for a web-based management UI. I generally try to minimize the amount of manual permissions tweaking I do in the system, as it feels a bit risky, partly because of issues with the database noted above. Managing users and groups in Plone is no better or worse than most decent content management systems or complex web applications, which is to say that it’s serviceable but not great. Again, it seems that this aspect of the system could benefit from a GUI app, or perhaps the web UI just needs more development, maybe adding some Ajax functionality to reduce page loads, etc.  A major positive development in user/group management has been the maturation of the Pluggable Authentication Service (PAS) and its integration into Plone.  Since I have a requirement to use an external authentication system (Shibboleth) via my Apache front-end, I started in the pre-PAS days by using a product called RemoteUserFolder, which worked well with Zope, but did not integrate with Plone. Now I use apachepas and AutoMemberMaker (I just learned that these products have been replaced by WebServerAuth). Since PAS is now the default user folder in Plone, these products integrate seamlessly and they have proven solid.

Continued in Part 3.

, , , ,

Leave a comment

My life with Plone, part 1

This is a statement of personal experience, not a manifesto. It’s likely to contain errors of both fact and judgment.  I’m just throwing it out there FWIW.

I started using Zope early in the 2.6 series, skinned a CMF site, then moved to Plone 1.x.  I built an intranet on Plone 2.0.5 and have upgraded that system through 2.5.3, 3.0.6, and the most current 3.1.x releases. Over that time, here’s what I’ve found.

Documentation

I really value well-organized clear and concise documentation aimed at what I would call an “intermediate” level of understanding. While there have been gradual improvements in this area, I still find that it can be a lot of work to get the information you need to solve a problem. Between books, plone-users, chat, plone.org, and miscellaneous web articles, there’s a lot out there, but navigating and sifting through it can be a frustrating and time-consuming process. I also use DocFinderTab and occasionally even the Zope 2 API documentation (which AFAICT hasn’t changed since I started using it).

Support for Previous Releases

I have found that, compared to other software pojects, Plone demands more frequent updating in order to stay on a “supported” version. As of today (9 Dec 08), 3.1.7 is the current stable release and 3.1.5.1 is the lowest supported version. While the new buildout procedure makes upgrading much easier, I’d prefer not to have to upgrade every couple of months.  Speaking of which …

Upgrading

While upgrading has become progressively less painful, especially if you do it frequently, there have been some major bumps in the road. 2.0.5 to 2.5.3 (migrating to Archetypes-based content types) was excruciatingly painful, and 2.5.3 to 3.0.6 was also challenging.  Since then it’s been easier, but it continues to feel somewhat risky. Customizations of default Plone skin files are susceptible to breaking and there’s not usually an easy way to know where this will happen except by brute force testing. My sense is that Zope 3-style customization will offer an improvement here, but my first casual attempts to understand how to migrate my TTW customizations caused my brain to hurt.

Customization

One the cool things about Zope 2 and the CMF upon which Plone was/is built is the acquisition machinery which supports the “layering” of “skins”.  Combined with TTW editing and in a few of clicks you’ve customized the interface.  With Plone 3 moving more and more to Zope 3 technologies, it seems to me that customization is getting harder for the “casual” developer. While I appreciate that this movement solves certain problems of acquisition and TTW customization, I worry that moving customization more into Python code and generally increasing the complexity of the system will mean that you won’t be able to do much with a Plone site without a Python programmer.

Backup/Restore and Versioning

My sysadmin loves the repozo script and I can’t think of a serious problem I’ve had with the ZODB. It’s been solid. OTOH, restoring specific content items from a backup to production is not as straightforward as it is with a file system or RDBMS. Within the Plone UI, the content versioning provided via the “History” tab has some advantages over the old “Undo” functionality, but the comparison feature only supports content diffs and doesn’t indicate changes in metadata, related content, etc.  And Undo did provide a way to “undelete” a deleted item (even though it didn’t/couldn’t always work), which has now been lost.

Continued in Part 2.

, , ,

Leave a comment