h1

Research Career Perspective

March 23, 2018

I was asked to do a general review of my research over the last decade or two, or three, or four … Gawd how long is it.  Lets just say I remember loading punch cards and tape drives, and being the first guy in the computer science class to use a … wait for it … a full screen edit called “vi” … because when I started everyone used a line editor called “ed”.  Students still remark how I effortlessly switch between “vi”, “emacs”, and “perl -pi -e” during work.  Not sure that’s good.

So the slides for the talk are here.    There is a lot more I could have said.

 

h1

What’s Hot in IT

March 2, 2018

Attended the Whats Hot in IT event held by the Victorian ICT for Women group.  See their Tweets about the event.  Prof. Maria Garcia De La Banda (2nd from left) gave a fabulous overview.

VICT4W

h1

Trying out DataCamp this semester

February 21, 2018

Our Master of Data Science students explore a lot of things and discuss.  I got a lot of requests to include the excellent material from DataCamp:

DataCamp logo

DataCamp – who support data science education for free

So we’ll see how it goes.  Not sure how well I’ll get to integrate it, because this semester I’m working more on our introductory statistics class.

h1

Whither the Scientific Method

January 28, 2018

Long before the Industrial Age in Europe we had the Dark Ages. Popular culture tells us it was believed that the Earth was flat, witches caused the plague, and the ways of the world were decreed by kings, or God himself. While rationalist explanations of the world appeared independently in many ancient civilisations, the scientific method as we know it became prominent in the 19th century as a remarkable series of scientific and engineering discoveries propelled the world into the industrial age. Indeed, Karl Pearson stated “the scientific method is the sole gateway to the whole region of knowledge.”.

With the pre-eminence of science in our modern society, controversies about science often occur in the media and public discussion, and the list of such areas is large. It doesn’t help that aspects of society, politics or religion have been falsely dressed up with “science,” so-called Scientism.  The expression “the science is settled” is a phrase from global warming skeptics that seeks to align global warning views with scientism (i.e., science is never settled so how can global warming be settled).  Note we can also view the statement “the science is settled” as a Socratian noble lie, therefore justify its use in public discussion.

So apart from false applications of science, i.e., scientism, what flaws with the scientific method are there?

Flaws in the Science?

Medical science has suffered bad press in recent times. Best known through the popular work of John Ioannidis, provocatively titled “Why Most Published Research Findings Are False”, testaments from famous and authoritative medical researchers about the flaws of published medical research abound. As an empirical computer scientist, I can assure you flaws in research are not restricted to medical science, its just that medical science is perhaps our most societally important area of science.

Some of the discussed flaws in research are the misuse of p-values though a variety of means. For an entertaining example, see John Bohannon’s “I Fooled Millions Into Thinking Chocolate Helps Weight Loss.” Other flaws are so-called surrogate endpoints (a biomarker such as a blood test is used as a substitute for a clinical endpoint such as a heart attack), and others still are poorly matched motivations, i.e., for academics, the idea of “publish or perish” but for industry this would be “publish and profit”.  Many lists of flaws have been published.

In all, however, the scientific method holds up as a valid approach because the flaws invariable amount to corruption of the original method. One way the medical community addresses this is by adding an additional layer on top of the standard scientific method, often called the systematic review.  This is where unbiased experts review a series of scientific studies on a particular question, make judgments about the quality of the scientific method and the evidence, and develop recommendations for healthcare. The systematic review is, if you like, quality control for the scientific method.

The End of Theory?

Another seeming assault on the scientific method comes from data science. In 2008 Chris Anderson of Wired published a controversial blog about “The End of Theory”. The idea is that the deluge of data completely changes how we should progress with scientific discovery. We don’t need theory, he claims, we just extract information from the deluge of data. The responses, and there are many, came quickly, for instance Massimo Pigliucci said “But, if we stop looking for models and hypotheses, are we still really doing science?”, and others questions the veracity and appropriateness of much observational data, and hence its suitability as a subject of analysis.

Anderson’s “end of theory,” like John Horgan’s “end of science”, is not so much wrong as much more complex that it first seems. The relationship between data science and the scientific method is not simple. To understand this, consider that the poster child for 19th century science was physics.  Physics, a mathematical science, is fundamentally different to say modern medicine. In physics, Eugene Wigner’s notion of the “unreasonable effectiveness of mathematics” holds sway: from a concise theory we can derive enormous consequences. A relatively small amount of well chosen scientific hypotheses have uncovered vast regions of the engineering and physics universe. For instance, weather predictions are currently based on simulations built using the Newtonian laws of physics coupled with geophysical and weather data.

The imbalance (a small number of scientific hypotheses needed to justify a large area of science) indeed suits the scientific method. Peter Norvig, however, points out this is not feasible in areas such as biology and medical science, where the unreasonable effectiveness of mathematics does not hold. In these areas, the complexities of the underlying processes means we cannot necessarily simulate the impact of a eating raw cocoa or drinking red wine on heart health because the simulation or derivations from fundamental properties of nature are just too complex.

Norvig’s colleagues at Google, some of the founders of data science, instead refer to the unreasonable effectiveness of data. That is, fundamental complexity of some sciences mean we should instead be using data-driven processes for discovery of scientific details.

Data Dredging

To understand how data science can change the scientific method, we need to look at how it should not change it. Statisticians like to talk derisively about data dredging, with p-hacking being the best known example. As in the chocolate study mentioned above, this is where studies are repeated (in some way) until a significant p-value is obtained. They argue data driven discovery is dangerous. But this is the wrong viewpoint for data science. In complex areas like medical science, we have many possible hypotheses and our intuitions can be poor in the complex world of biology.

Computer science has an elegant theory of complexity called NP-completeness, which is the notion that one may need to test an exponential number of things before finding one that works. This indeed is the situation we find ourselves with hypothesis testing in the broader scientific world where the unreasonable effectiveness of mathematics fails.

In the early days of machine learning I worked at Prof. Ross Quinlan’s lab in Sydney. We soon discovered our own version of Ioannidis’ flaws in medical science that applied to machine learning. We called it theory overfitting, in contrast to regular overfitting which is an artifact of the bias-variance dilemma in statistics and machine learning. People tested a bunch of different theories on a small number, say 5 data sets, and eventually they find one that works, and so write it up and publish it. In truth its just another variant of p-hacking.

In data science, if we’re appling machine learning or neural network algorithms to a body of data, we are invariably trying to solve an NP-complete problem and are thus subject to overfitting or p-hacking. Even if we employ careful statistical methods to try to overcome this, we may subsequently be doing theory overfitting. However, if we don’t employ machine learning methods, we may never uncover reasonable hypotheses in the exponential pool of candidates. This is the conundrum of data science for the scientific method when used in broader non-mathematical domains.

Powering the Scientific Method

Organisations and hard nose businesses have this conundrum effectively solved. At Kaggle for instance, and in TREC competitions, a test data set is always hidden from the machine learners, and only used for a final validation, which acts like a (final) cycle of the scientific method. The initial “develop a general theory” step of the scientific method has been done with machine learning. This can be considered to be millions of embedded hypothesise-test cycles. Thus we have an epicycle view of the scientific method.

But applying this approach in the medical world is not straight forward.  The medical research world keeps data registries that feasibly can be used to obtain data for discovery purposes. However, to obtain data, one usually has to apply for ethics clearance/approval, that the intended use of the data is good. The ethics committees who oversee the approval are the gatekeepers of data, and oftentimes they expect to see a valid scientific plan, not an open-ended discovery proposal. With the epicycle view of the scientific method, registries, when they release data for discovery exercises, would need to withhold some data for a final validation step in order to preserve the validity of the scientific method.

h1

To good health!

January 10, 2018

So enrolment sessions start soon for our incoming Master of Data Science students.  I know its a stressful time for some students in terms of “life”.  I usually talk briefly about staying healthy, and Monash offers various services to support this.  But for PhD students I think its important to take this on as a lifestyle objective.  They are undertaking a knowledge intensive career path and brain health will be critical for their future career.

Disclaimer:  Now, this page is full of opinions and pointers to, in some case, controversial material.  I’m just a little old computer science professor, so my opinions have no real backing, and I have no recognised expertise. All care but no responsibility for what I say! 

The fact is, keeping healthy and understanding how to keep healthy in the modern world is a subject fraught with challenges.  To understand this, consider the following:

  1. The official Australian government position on colds and flu prevention, and the official USA government position:  hygiene and vaccines.  What’s missing:  discussion of healthy diet, exercise and other lifestyle factors to strengthen and repair the immune system.
  2. The Time magazine reports extensive research shows vitamin D helps prevent colds and flu, so some sunshine is also important.  No mention of this in the official government positions above!
  3. Believe it or not, in the USA prescription drugs are the third leading cause of death!  There is a larger issue here in that most published medical research findings are false.  Note, I see this is a systemic thing not limited to medical research.  However, the medical research community has extensive, concerted efforts like systematic reviews to address the issue.
  4. Tobacco science is a term used to describe fake science protecting an industry.  Read about the Tobacco Institute and see the movie The Insider.  How much of this goes on in the food and drugs industry?  Lots according to Nina Teicholz, see point 8 below.
  5. Sugar is now known to be very damaging to the health.  Here is a hard hitting discussion about it, though note quite a few of these claims are considered controversial.  But it is known that sugar suppresses the immune system.  Figuring out your sugar consumption is challenging.   There are rumors (in movie form) of tobacco science going on here too.
  6. Energy drinks rot the teeth, like soft drinks.  Its due to the high acid content.  Its certainly not clear they give any energy.
  7. Artificial sweeteners are not a substitute, in fact evidence suggests they have poor health impacts, and they mess up the brains analysis of your food intake.
  8. Fats are the subject of a massive onslaught from advertisers.  For years we were told to avoid butter and use margarine instead, but now it seems butter is good.    The rather hilarious and utterly confusing history of health advice about butter is in this Butter Studies Roundup. Current conflicting advice is now being broadcast about the humble coconut.  See Nina Teicholz talk about the complex history of our understanding of fats in her TEDx seminar, based on her best selling book, “The Big Fat Surprise”.
  9. The health of organic produce is currently a propaganda battleground.   None other than former tobacco scientist Henry I Miller (he was a founder of TASSC) has claimed its an expensive scam.  Hint:  organics are also lower in toxic pesticide residue, which is why I would get them.
  10. The commercial world has taken on healthy eating big time, and it is the fastest growing segment of the food industry.  Monash University has done a wonderful job of getting really good fast food vendors at the Caulfield campus food court.  If you’re an old time traveller, you should also be aware of the huge changes in airport food courts and tourist spots like London when it comes to healthy eating options.

Summary:  There is lots of conflicting and bad advice out there!  Heck, even the government websites seem to have errors of omission.

Now, if we consider the specific position of someone who wants their brain to function well, then consider the following:

  1. Short term exercise is known to boost mental performance.
  2. Meditation and mindfulness is also known to boost performance in exams.
  3. Long term sitting is considered to be as bad for health as smoking!  Here is a poster of the dangers.
  4. There are also lifestyle recommendations about studying from scientists:  don’t cram for subjects, learn slowly over the semester.
  5. Recent studies show the brain can be encouraged to grow new cells.
  6. The brain is mostly fat, so we need healthy fats to work well.  Don’t believe a lot of what you read about fats!  Cholesterol is also important for the brain.
  7. Sugar consumption (e.g., soft drinks, commercial juices, commercial cereals, flavoured yogurts, etc. etc. etc.) is bad for the brain, as well as the immune system.
  8. Canola oil is bad for the brain.  This one is important because most cheap salad oils, margarines and many food products are loaded with it.
  9. All sorts of food and chemicals are bad for the brain.  Here’s a TEDx talk on details.  Note TEDx means not official (is this a reliable information source?).
  10. Deep sleep is the basis for memory, learning and health.   In particular, without deep sleep, your brain will not be functioning properly and your memory will be impaired.  Here is a disquieting Google talk on health and sleep (along the lines of the hideous anti-smoking adverts some countries have), but there are many more on this.
  11. Adult neurogenesis is the process by which we adults gain new brain cells.  Not surprisingly, this is very popular amongst the Silicon Valley crowd, and I suspect is also a domain where snake-oil salesman like to peddle.  Nonetheless, here is a video on it:  a TED talk.

Note, for each of these, there are 10’s-100’s of good articles and scientific literature to back it up, though oftentimes conflicting scientific literature as well.  I’m just giving generally readable and somewhat respectable accounts.  A lot of these issues remain controversial, and possibly there is some tobacco science going on, but its hard for us non-experts to really know.

Anyway, I hope from this you understand the complexity of trying to stay health, and trying to keep your brain functioning well in the modern world.

I’m probably a bit extreme but I say,

About eating and food:

  • Benjamin Franklin said “An ounce of prevention is worth a pound of cure.” and my Dad lived by it.  I agree heartily.   So eating well and living a healthy lifestyle is better than loading yourself up with drugs to maintain performance.

  • Try and cook your own meals from real ingredients.  After a while, it becomes easy and its a great way to wind down with friends.
  • If someone’s great grandmother (anyone’s, Fiji, Vietnam, Sweden, …) didn’t make the food 100 years ago, its probably not good for you.
  • Don’t take dietary or health advice from Big Food.  In fact, looking at the government advice (listed above) on the flu, I’d say their’s is missing some major points too for some issues.
  • Try and avoid packaged meals, fast food, and canned and bottle drinks.  Likewise, avoid most commercial fruit juices which have way too much sugar and have lost too much of the fabulous nutrients in the original fruit due to being pasteurised or reconstituted.
  • Go low sugar, low refined carbohydrates and healthy fats.  Its a lifestyle thing, not a diet.  Once you do, you’ll discover all the amazing subtle flavours you’ve missed from traditional foods and realise how horrible standard breads, sweet deserts, snack bars and cakes really are:  the sugar masks the real flavour, and it gives you a longer term bad after taste, and refined carbohydrates have removed a lot of the flavours.
    • Healthy fats is challenging to maintain because Big Food likes to put unhealthy canola oil in everything:  most salad oils, hummus and deli mixes are mostly canola oil, as is margarine.
    • Well made healthy bread is truly remarkable in flavour, but it costs more and you cannot find it at the big chains.  Have it with a thick spread of (grass-fed) Tasmanian or New Zealand butter.  Nothing better!
  • Just avoid artificial sweeteners.  Once you’ve gone cold turkey and got off the sugar addiction you wont be craving sugar and you’ll feel better for it.
  • Health slogans on food products, “low fat”, “low cholesterol” often mean its bad for you!   Low fat usually means high sugar, for instance.

About other aspects of health:

  • Get exercise, and make it a lifestyle thing.  When you’re older, you’ll discover you cannot function well as a knowledge worker without it.
  • Don’t sit at your desk for long hours.  You need to get up and move around every hour!  Also, become aware of your posture.   Don’t become a hunchback!  Some 2nd years are already heading that way.
  • When you’re mentally worn out, a quick nap or a brisk walk does wonders, and both have scientific backing.
  • Make sure you are getting proper sleep.  That can mean organising your assignments and study properly so you don’t need a to do a bunch of all-nighters to get through.  But it also means setting up the right environment at home for sleep.
  • I know of few cases where drugs and alcohol support good health or brain functioning, including many so-called smart drugs.  Most are dangerous to the liver, as are many medicines.  Headache and pain medicine is far more dangerous and damaging than many other things!  Good food, exercise and sleep is how you increase performance.
  • Note there is a whole field of nootropics which is emerging as an alternative therapy (not condoned by medical science).  Human biology is extremely complex, and the scientific method is fairly crude as a developmental model of knowledge, especially when its constantly interfered with by commercial interests.  I expect some good natural supplements to eventually emerge here where they naturally enhance human biology.
  • Routine … that’s what the body needs.  For sleep, for eating, for study, for exercise, routine is critical part of making it function well.

Anyway, I have said too much already.  In case you’re wondering, I wrote this initially while on holidays.  No time for a Data Science professor to talk about this stuff during semester!   But I’ve been updating it ever since.  Students badly need some good advice in this area.  The modern world is a minefield for those wanting to stay healthy.

But bear in mind, I have no qualifications or expertise when it comes to health.  These are mere opinions!

h1

Picking Conferences

January 7, 2018

As a PhD student starting out, you do have some career options.  Likewise, as a typical junior academic, with limited budget and research time, you have similar career options.  A main one which I’ll discuss here is:  Which conference(s) should I got to?  This is peculiar to computer scientists whose conferences are competitive publications (say 20-25% acceptance rate) and count as publications.

So you only get time to attend a few conferences.  Likewise, you only get time to write papers for a few.  So you want them to count.  Conferences each have their own style.  Best way to think of it is that a conference is a tribe where membership is part-time.  You have to take time to learn about the habits and preferences of the tribe, i.e., in terms of paper content.  If the tribe always starts off with 20% of detailed theoretical definitions then you have to as well.  If they do certain kinds of experiments, then so should you.  Think of these sorts of things as tribal markings.  To be innovative, you generally need to do so from inside the system.  I know this sounds conformist, and belief me, I am completely non-conformist myself, but generally its how conferences work, largely as a result of the reviewing system. If a trusted member of the tribe starts quoting classical, venerated philosophers, so will the others.  If a complete unknown person submits a paper quoting venerated philosophers, then it’ll be viewed as weird unless they have enough other tribal markings on their work to accepted.

I have a number of conferences I really like where I understand the general tribal markings and am happy to live with them.  So SIGKDD has solid experimental work, ICML has innovative new methods, ACL has applications of machine learning to real linguistic problems.  They sometimes have additional tribal markings that can be more or less problematic.

Anyway, as a junior academic, you have to target a few conferences and learn to become a reliable tribal member.  You might want to pick a few authors and build on their work.  Or you might want to pick a specialised problem.  Regardless, to publish in particular venues you’ll have to get to know the tribal preferences and adhere to them.  Doing good research is one thing, and really good research will usually speak for itself, but if your contribution is not outstanding, say “merely” at the top 25 percentile of work, then you have to follow the tribe to be accepted into the tribe.  That takes time.

Moreover, the vibe at the conference is always much, much more than the printed proceedings.  You need to be there:  hear the questions, watch the audience, chat to others in the breaks, see the quality of the presenters.  What is important and influential?  What is losing out, perhaps because it was trendy rather than productive?  All this happens at the conference.  You need to be there to see it.  Otherwise, you’ll be a year behind the others … new ideas for next year’s conference are often the germ of an idea at this year’s conference.  Moreover, it always helps to see the movers and shakers in action.  What sort of people are they?  How do they present their work?

So what does this mean to the junior academic?  You need early on to target a particular conference, subject or influential author’s/group’s body of work, and learn what it is they do.  That’ll take time.  So if you don’t see yourself as being involved in that community 5 years down the track, you probably shouldn’t be making that effort.  If you think their research doesn’t have a good future, then again, you probably shouldn’t be making that effort.  Pick some conferences with this in mind, and try and go along semi-regularly to keep track of things and pick up the vibe.

h1

Asian Conference on Machine Learning

November 11, 2017

Heading off to ACML in Seoul to present the paper “A Word Embeddings Informed Focused Topic Model” for PhD student He Zhao.  He is off elsewhere, at ICDM in New Orleans presenting another paper, “MetaLDA: a Topic Model that Efficiently Incorporates Meta Information”.  The MetaLDA algorithm incorporates Boolean side information, beating all others, and the newer WEI-FTM algorithm incorporates general side information but as a focused topic model.  He is a prolific coder, with some of his work on Github.

ACML is getting to be a great conference.  Always great invited talks and tutorials.  A worthy end of semester break for me.

Abstract
In natural language processing and related fields, it has been shown that the word embeddings can successfully capture both the semantic and syntactic features of words. They can serve as complementary information to topics models, especially for the cases where word co-occurrence data is insufficient, such as with short texts. In this paper, we propose a focused topic model where how a topic focuses on words is informed by word embeddings.  Our models is able to discover more informed and focused topics with more representative words, leading to better modelling accuracy and topic quality. With the data argumentation technique, we can derive an efficient Gibbs sampling algorithm that benefits from the fully local conjugacy of the model.  We conduct extensive experiments on several real world datasets, which demonstrate that our model achieves comparable or improved performance in terms of both perplexity and topic coherence, particularly in handling short text data.
Keywords: Topic Models, Word Embeddings, Short Texts, Data Augmentation