Where is the best place to store images on your Web site? If you have a huge content application and store all your images in a single folder called “images”.you might land up you application in “BLack Hole” situation..

Storing all the images of the application in just one folder have many problems like.

1. The “black hole”.
2. Name conflict.
3. Access control.
4. Scalability.
5. Performance.

The Black Hole:

“Black hole” is an appropriate name for the  folder where all the files are stored , because files go into this folder but never come out. Over time, more and more new files are added to the “Black Hole” folder. However, when users are deleted files that are no longer useful are left in the folder. Why? The primary reason is because files stored in the  folder may be  shared resource and it’s hard to tell if a given file is being used or not . So the “Black Hole” folder is left unmanaged and becomes a stale collection of any file that has ever been used on the application.

Name Conflict:

One of the first problems content management encounters is file name conflict, when a user uploads a files with a file name that already exists in the repository.Most of the management systems append ‘_1′ , ‘_2′ to the file name to remove the conflicts but that looks very ugly for the system as well as users.The more file you have in a single folder, the greater the chances that a name conflict will occur.

Access control:

Your management team may control access to the  documents/files .Using Singe folder to store all the files will cause the access control conflicts. Content authors are permitted to delete, rename or replace images in that folder. Content creators can manipulate images used by documents that they don’t have access to, and can even change the branding of the application.

Scalability:

Most application provide an file library interface that either displays a list box or thumbnails of files in the library. If the  folder contains hundreds or thousands of files , it becomes impractical for applications to present these images in this way to content authors and for content authors to find the appropriate files.

Performance:

File systems don’t perform well when a large number of files are stored in a single folder. The file management library may open slowly for users, and if the  library displays metadata about file such as dimensions, file size or date last modified, this slows performance even further. Large  libraries also require more data to be transmitted over the network, which can be resource intensive and slow. 

A better way to manage Files

In this article, we will limit the discussion to storing images on the file system, although this approach can be adapted to storing images in the database as well.

The solution to the problems created by using a single folder can be addressed by creating a different folder on the file system for each User/Object. The folder name should be the ID of the User/Object in the application and all the files used by a given User/Object should be stored in this folder. Although these folders can be created anywhere on your site, the examples in this article will assume all these folders are sub-folders stored in a folder called “User_Files” of the root of the application.

As discussed previously, file systems don’t perform well when a large number of files are stored in a single folder. The same is true for sub-folders. The solution is therefore to create grouping folders that will significantly reduce the number of immediate sub-folders in any given folder.

If your system uses sequentially numbered User/Object IDs, then you can use a math.fmod(x, y) arithmetic operation to create the name of the grouping folders. (x is your User/Objects ID and Y is your max folder limit number lets say 999). fmod operations return the remainder from a division of two numbers. So to get 999 grouping folders, simply fmod the User/Object ID by 999, which will produce numbers from 0 to 999, which you can use to name your grouping folders. The following illustrates how these folders will be nested:

• User_Files
  • /1     #fmod(objectID,999)
    • /1000    #objectID
      • /my_resume.pdf   #FileName
      • /project_details.doc
    • /1256
      • /NDA.pdf
  • /337
    • /2335
      • /company_policy.docx
      • /designs.psd
    • /1257
      • /my_image.gif
    • /1513
  • /338
    • /2336
      • /logos.jpg
      • /product1.tiff
      • /HomePage.jpeg

Conclusion

Storing files  in separate folders solves many of the problems caused by using a single folder. This approach is simple and reliable and can  be used for attachments (documents like PDF, Microsoft Word, Zip files, etc.). In fact, you can store both images and attachments in the same folder.

This article shows how to get Solr index and provide search results for a simple MySQL table.  The search results will be provided in XML/JSON, so you can get your web application to read the results, parse it and display it any form desired.

For this article, we will use the solr 3.6.1 build (17-July-2012).You can download the  Solr from Download latest Solr Release. As we are going to use Direct database and XML import through Solr’s DataImportHandler (the DIH).you also need to download the latest MySQL JDBC driver from http://dev.mysql.com/downloads/connector/j/

Install Solr

1. Unzip or untar the downloaded solr  build package. Assume the unzipped directory is solr.
2. cd into solr/example directory which has a standalone solr server running on jetty.
3. Execute the server by “java -jar start.jar”
4. Test the server by accessing http://<servername_or_ip&gt;:8983/solr/admin/ .  If you get an admin page with a search box, then your solr is running well, and ready.

Configure MySQL Database

1. Copy the downloaded mysql jdbc driver file into solr/lib directory.
2. Create a new xml file called data-import.xml , change the obvious variables to suit your DB.  In this example, I am indexing a example DB table called jos_content.

<?xmlversion="1.0"encoding="UTF-8"?>
<dataConfig>
    <dataSource type="JdbcDataSource"
                  driver="com.mysql.jdbc.Driver"
                  url="jdbc:mysql://localhost/database"
                  user="user"
                  password="password"/>

    <document name="doc">
        <entity name="jos_content"
                  query="select * from jos_content WHERE state=1"
                  deltaImportQuery="SELECT * FROM `jos_content` WHERE id='${dataimporter.delta.job_jobs_id}'"
          deltaQuery="SELECT id FROM `jos_content` WHERE modified > '${dataimporter.last_index_time}'">

            <field column="id" name="id" />
            <field column="title" name="title" />
            <field column="introtext" name="introtext" />
            <field column="fulltext" name="fulltext" />
        </entity>
    </document>
</dataConfig>

3.  Edit file solrconfig.xml which is located in solr/example/solr/conf directory. Add the following requestHandler entry if not already existing.

<requestHandler name="/dataimport">
    <lst name="defaults">
      <str name="config">/solr/data-config.xml</str>
    </lst>
  </requestHandler>

4. Now we will configure solr’s schema by editing schema.xml in solr/example/solr/conf directory. Add or edit the following fields as required. The xml format is self explanatory.

<fields>
        <field name="id" type="string" indexed="true" stored="true" required="true"/>
    <field name="title" type="text" indexed="true" stored="true" required="true"/>
    <field name="introtext" type="text" indexed="true" stored="true" required="true"/>

    <field name="fulltext" type="text" indexed="true" stored="true" required="true"/>
    <dynamicField name="*" type="ignored" />
 </fields>

 <uniqueKey>id</uniqueKey>
 <!-- field for the QueryParser to use when an explicit fieldname is absent -->
 <defaultSearchField>fulltext</defaultSearchField>

5. Stop and restart solr instance. Check if there are any jdbc errors, this could happen if the jdbc drivers are not properly installed.

Performing full or delta indexing

If everything works correctly, you can get solr to fully index the configured tables by accessing the following command via your browser. http://<your_solr_server&gt;:8983/solr/dataimport?command=full-import
You can check the status of the command by accessing
http://<your_solr_server&gt;:8983/solr/dataimport
If everything works correctly, you can now search for data from http://<your_solr_server&gt;:8983/solr/admin/ and you should now have data results in XML format.

To do an incremental or delta indexing of data since the last full or delta, increment, issue the command
http://<your_solr_server&gt;:8983/solr/dataimport?command=delta-import
You can now access these xml results from your web application.  There are client api’s available for python,RoR, php, java etc.

References

http://wiki.apache.org/solr/DataImportHandler#head-df246a3aed0bb38297f3449bc35a0bdf38a272b5

http://lucene.apache.org/solr/tutorial.html

# coding=utf-8

class Check_Credit_Card(object):
    """
    Class to check either the Credit card number is valid or not
    """
    cc_patterns = dict(amex={15: ('34', '37')},
        carteblanche={14: ('300', '301', '302', '303', '304', '305', '36', '38')},
        dinersclub={14: ('300', '301', '302', '303', '304', '305', '36', '38')}, discover={16: ('6011')},
        enroute={(15): ('2014', '2149')}, jcb={15: ('2131', '1800'), (16): ('3')},
        mastercard={(16): ('51', '52', '53', '54', '55')}, visa={(13, 16): ('4')})

    def check(self,cc_num,cc_type):
        """
        Main function to check the card
        """
        cc_type = cc_type.strip().lower().replace(' ', '')
        cc_num = str(cc_num)

        if cc_type in self.cc_patterns.keys():
            # Perform length check
            valid_lengths = self.cc_patterns[cc_type].keys()
            if len(cc_num) not in valid_lengths:
                return (False, 'Invalid length (%s) for %s' % \
                    (str(valid_lengths).strip('[]'), cc_type))

            # Perform prefix check
            valid_prefixes = self.cc_patterns[cc_type][len(cc_num)]
            prefix_match = False
            for prefix in valid_prefixes:
                prefix_length = len(prefix)
                if cc_num[:prefix_length] == prefix:
                    prefix_match = True
            if not prefix_match:
                return (False, 'Invalid prefix (not in %s)' % \
                    ','.join(valid_prefixes))

            # Perform mod 10 check
            if cc_type == 'enroute':
                pass # Only cc_type that doesn't do a mod 10 check
            else:
                check_digits = []
                is_even = False
                for digit in cc_num:
                    if not is_even:
                        check_digits.append(int(digit))
                    else:
                        mult_digits = str(int(digit) * 2)
                        check_digits.append(mult_digits[0])
                        if len(mult_digits == 2):
                            check_digits.append(int(mult_digits[1]))

                check_sum = sum(check_digits)
                if check_sum % 10 != 0:
                    return False, 'Invalid check sum (%d)' % check_sum

            return True, ''
        else:
            return False, 'Invalid card type (%s)' % cc_type

if __name__ == "__main__": #For testing you can Use the file directly in python console..
    x = Check_Credit_Card()
    x = x.check('5235235879456682','MASTERCARD')
    print x[0]

#VIEWS.PY
def Check_Validity(request):
    if request.method == 'POST':
        Data = request.POST.copy()
        cc_obj = Check_Credit_Card()
        result = cc_obj.check(Data.get('card_no',''),Data.get('card_type',''))
        return HttpResponse(simplejson.dumps({'result':result[0],'reason':result[1]}), mimetype="application/json")
    return HttpResponse(simplejson.dumps({'result':'Not a valid method'}), mimetype="application/json")

#URLS.PY
url(r'^check_validity$',Check_Validity,name="Check_Validity_View")

import SimpleHTTPServer
import SocketServer

PORT = 8000

class SimpleServer():

    #Simple Http server python usage
    def start(self):
        Handler = SimpleHTTPServer.SimpleHTTPRequestHandler
        httpd = SocketServer.TCPServer(("", PORT), Handler)
        print "serving at port", PORT
        httpd.serve_forever()

if __name__ == "__main__":
    SimpleServer.start()

# Area of Polygon using Shoelace formula
# http://en.wikipedia.org/wiki/Shoelace_formula
# FB - 20120218
# corners must be ordered in clockwise or counter-clockwise direction
def PolygonArea(corners):
    n = len(corners) # of corners
    area = 0.0
    for i in range(n):
        j = (i + 1) % n
        area += corners[i][0] * corners[j][1]
        area -= corners[j][0] * corners[i][1]
    area = abs(area) / 2.0
    return area

# examples
corners = [(2.0, 1.0), (4.0, 5.0), (7.0, 8.0)]
print PolygonArea(corners)
corners = [(3.0, 4.0), (5.0, 11.0), (12.0, 8.0), (9.0, 5.0), (5.0, 6.0)]
print PolygonArea(corners)

Source (Python recipe)

from math import sin, cos, radians, acos

# http://en.wikipedia.org/wiki/Earth_radius
# """For Earth, the mean radius is 6,371.009 km (˜3,958.761 mi; ˜3,440.069 nmi)"""
EARTH_RADIUS_IN_MILES = 3958.761

def calculate(lat_a, long_a, lat_b, long_b):
    """all angles in degrees, result in miles"""
    lat_a = radians(lat_a)
    lat_b = radians(lat_b)
    delta_long = radians(long_a - long_b)
    cos_x = (
        sin(lat_a) * sin(lat_b) +
        cos(lat_a) * cos(lat_b) * cos(delta_long)
        )
    return acos(cos_x) * EARTH_RADIUS_IN_MILES

#SFO (37.676, -122.433) to NYC (40.733, -73.917)
#calculate -> 2570.9028268899356
#A US government website (http://www.nhc.noaa.gov/gccalc.shtml) -> 2569
#This website (http://www.timeanddate.com/worldclock/distanceresult.html
# ?p1=179&p2=224), from which I got the SFO and NYC coordinates, -> 2577

but this did’t solve my whole issue ,in my request i have Zip Codes only so the crack was to get the database of all the zips from ZIP DIR in csv or sql format and add to the database of the App e.g the table name is ZipCode.

from django import template
from models import ZipCode

register = template.Library()

EARTH_RADIUS_IN_MILES = 3958.761

@register.filter
def calculate_distance(zip1,zip2):
    coordinate1 = ZipCode.objects.get(zip = zip1)
    coordinate2 = ZipCode.objects.get(zip = zip2)
    distance = calculate(coordinate1.latitude,coordinate1.longitude,coordinate2.latitude,coordinate2.longitude)
    if distance>= 0:
        return "%.2f"%round(distance,2)
    return "Sorry can't determine the distance from %s TO %s"%(zip1,zip2)

Usage

{% if request.GET.ZipCode %}
     {{ doctor.ZipCode|calculate_distance:request.GET.ZipCode }}
{% endif %}

def CalculateApacheIpHits(logfile_pathname):

    # make a dictionary to store Ip's and their hit counts and read the
        # contents of the logfile line by line

    IpHitListing = {}
    Contents = open(logfile_pathname, "r").readlines()

    # go through each line of the logfile
    for line in Contents:

                #split the string to isolate the ip
                Ip = line.split(" ")[0]

                # ensure length of the ip is proper: see discussion
        if 6 < len(Ip) < 15:
            # Increase by 1 if ip exists else hit count = 1
                        IpHitListing[Ip] = IpHitListing.get(Ip, 0) + 1

    return IpHitListing

# example usage
HitsDictionary = CalculateApacheIpHits("/usr/local/nusphere/apache/logs/access_log")
print HitsDictionary["127.0.0.1"]

This function is quite useful for many things. For one, I often use it in my code to determine how many of my “hits” are actually originating from locations other than my local host. This function was also used to chart which IP’s are most actively viewing pages that are served by a particular installation of Apache.

As for the method of “validating” the IP, it as is follows: 1) an IP address will never be longer than 15 digits (4 sets of triplets and 3 periods); 2) an IP address will never be shorter than 6 digits (4 sets of single digits and 3 periods). The whole purpose of this validation is not to enforce stigent validation (for that we could use a regular expression), but rather to avoid the possibility of putting blantently “garbage” data into the dictionary.

There are quite a few ‘enum’ recipes already here. This one is short, cheap, and has neither bells nor whistles

def enum(typename, field_names):
    "Create a new enumeration type"

    if isinstance(field_names, str):
        field_names = field_names.replace(',', ' ').split()
    d = dict((reversed(nv) for nv in enumerate(field_names)), __slots__ = ())
    return type(typename, (object,), d)()

You define an enumeration the same way as collections.namedtuple: with a type name, followed by a sequence of identifiers (it may be a string separated by commas and/or whitespace):

>>> STATE = enum('STATE', 'GET_QUIZ, GET_VERSE, TEACH')

It returns a (single) instance of a newly created type. The enumeration elements are integers, starting at 0, exposed as attributes. There is no way to change that: if you need specific values, look elsewhere. In fact, it is as if the enum were declared as:

# not actual code
class STATE(object):
  __slots__ = ()
  GET_QUIZ, GET_VERSE, TEACH = range(3)

STATE = STATE()

The enumeration elements are read-only:

>>> STATE.GET_VERSE
1
>>> STATE.GET_VERSE = 8
Traceback (most recent call last):
  ...
AttributeError: 'STATE' object attribute 'GET_VERSE' is read-only
>>> del STATE.GET_VERSE
Traceback (most recent call last):
  ...
AttributeError: 'STATE' object attribute 'GET_VERSE' is read-only

This recipe has been tested with Python 2.4 ranging up to 3.1

######################################################################
## 
## Feature Broker
## 
######################################################################

class FeatureBroker:
   def __init__(self, allowReplace=False):
      self.providers = {}
      self.allowReplace = allowReplace
   def Provide(self, feature, provider, *args, **kwargs):
      if not self.allowReplace:
         assert not self.providers.has_key(feature), "Duplicate feature: %r" % feature
      if callable(provider):
         def call(): return provider(*args, **kwargs)
      else:
         def call(): return provider
      self.providers[feature] = call
   def __getitem__(self, feature):
      try:
         provider = self.providers[feature]
      except KeyError:
         raise KeyError, "Unknown feature named %r" % feature
      return provider()

features = FeatureBroker()

######################################################################
## 
## Representation of Required Features and Feature Assertions
## 
######################################################################

#
# Some basic assertions to test the suitability of injected features
#

def NoAssertion(obj): return True

def IsInstanceOf(*classes):
   def test(obj): return isinstance(obj, classes)
   return test

def HasAttributes(*attributes):
   def test(obj):
      for each in attributes:
         if not hasattr(obj, each): return False
      return True
   return test

def HasMethods(*methods):
   def test(obj):
      for each in methods:
         try:
            attr = getattr(obj, each)
         except AttributeError:
            return False
         if not callable(attr): return False
      return True
   return test

#
# An attribute descriptor to "declare" required features
#

class RequiredFeature(object):
   def __init__(self, feature, assertion=NoAssertion):
      self.feature = feature
      self.assertion = assertion
   def __get__(self, obj, T):
      return self.result # <-- will request the feature upon first call
   def __getattr__(self, name):
      assert name == 'result', "Unexpected attribute request other then 'result'"
      self.result = self.Request()
      return self.result
   def Request(self):
      obj = features[self.feature]
      assert self.assertion(obj), \
             "The value %r of %r does not match the specified criteria" \
             % (obj, self.feature)
      return obj

class Component(object):
   "Symbolic base class for components"

######################################################################
## 
## DEMO
## 
######################################################################

# ---------------------------------------------------------------------------------
# Some python module defines a Bar component and states the dependencies
# We will assume that
# - Console denotes an object with a method WriteLine(string)
# - AppTitle denotes a string that represents the current application name
# - CurrentUser denotes a string that represents the current user name
#

class Bar(Component):
   con   = RequiredFeature('Console', HasMethods('WriteLine'))
   title = RequiredFeature('AppTitle', IsInstanceOf(str))
   user  = RequiredFeature('CurrentUser', IsInstanceOf(str))
   def __init__(self):
      self.X = 0
   def PrintYourself(self):
      self.con.WriteLine('-- Bar instance --')
      self.con.WriteLine('Title: %s' % self.title)
      self.con.WriteLine('User: %s' % self.user)
      self.con.WriteLine('X: %d' % self.X)

# ---------------------------------------------------------------------------------
# Some other python module defines a basic Console component
#

class SimpleConsole(Component):
   def WriteLine(self, s):
      print s

# ---------------------------------------------------------------------------------
# Yet another python module defines a better Console component
#

class BetterConsole(Component):
   def __init__(self, prefix=''):
      self.prefix = prefix
   def WriteLine(self, s):
      lines = s.split('\n')
      for line in lines:
         if line:
            print self.prefix, line
         else:
            print

# ---------------------------------------------------------------------------------
# Some third python module knows how to discover the current user's name
#

def GetCurrentUser():
   return os.getenv('USERNAME') or 'Some User' # USERNAME is platform-specific

# ---------------------------------------------------------------------------------
# Finally, the main python script specifies the application name,
# decides which components/values to use for what feature,
# and creates an instance of Bar to work with
#
if __name__ == '__main__':
   print '\n*** IoC Demo ***'
   features.Provide('AppTitle', 'Inversion of Control ...\n\n... The Python Way')
   features.Provide('CurrentUser', GetCurrentUser)
   features.Provide('Console', BetterConsole, prefix='-->') # <-- transient lifestyle
   ##features.Provide('Console', BetterConsole(prefix='-->')) # <-- singleton lifestyle

   bar = Bar()
   bar.PrintYourself()

#
# Evidently, none of the used components needed to know about each other
# => Loose coupling goal achieved
# ---------------------------------------------------------------------------------

Inversion of Control (IoC) Containers and the Dependency Injection pattern have drawn some attention in the Java world, and they are increasingly spreading over to .NET, too. (Perhaps we are facing a sort of “Infection OUT of Control” – IooC?

IoC is all about loose coupling between components of an application, about cutting off explicit, direct dependencies, plus some goodies (most of which are useful in statically typed languages only, like automatic type/interface matching). A thorough discussion on the subject can be found at http://www.martinfowler.com/articles/injection.html .

In statically typed languages, an IoC container is quite a challenge. But at the heart of it, there are only few key concepts behind it.

Components do not know each other directly
Components specify external dependencies using some sort of a key.
Dependencies are resolved late, preferably just before they are used (JIT dependency resolution).
Dependencies are resolved once for each component.

You guessed it – it should not be such a big deal to do this in python!

And indeed, a combination of a broker, descriptors and lazy attributes brings about pretty much the same core result as those IoC containers – effectively in little more then 50 lines (not counting demo code, comments and empy lines).

So what does the code do?

It offers a mechanism to register provided “features”.
It offers a mechanism to “declare” required features in a readable way as attributes.
The required features are resolved (injected) as late as possible – at access time.
It provides for reasonable verification of injected dependencies.

The supported injection type is “Setter Injection”, which basically means that dependencies are expressed through attributes. There is another type of injection, the “Constructor Injection”, but that one builds heavily on static typing and can therefore not be employed in python (or at least I could not think of any elegant way to do it).

Of course, there are a million ways to enhance the code in this recipe. But on the other hand, isn’t that true for almost any piece of code?

Cheers and happy injecting!